Reporting health care decision models: a prospective reliability study of a multidimensional evaluation framework

Area of Science:

• Health Sciences
• Biostatistics
• Medical Modeling

Background:

• Quality assessment of scientific modeling studies is crucial.
• Existing frameworks may lack comprehensive evaluation criteria.
• The Phillips-checklist is a proposed tool for assessing modeling study quality.

Purpose of the Study:

• To evaluate the inter-rater reliability of the Phillips-checklist.
• To identify areas for improvement in the checklist for better quality assessment of modeling studies.

Main Methods:

• Six raters assessed nine modeling studies across three medical specialties.
• Inter-rater reliability was measured using Intra-class correlation (ICC).
• Raters provided qualitative feedback on their experience with the checklist.

Main Results:

• Overall inter-rater reliability was moderate (ICC = 0.69), with no significant rater effect (p=0.064).
• A significant rater effect was observed for clopidogrel studies (p < 0.001), potentially due to lower study variability.
• The checklist facilitated structured assessment, but some items required clarification, and rater variability was notable.

Conclusions:

• The Phillips-checklist offers a structured approach to assessing modeling studies but requires refinement.
• Improvements to specific scoring items are recommended to enhance clarity and reduce rater variability.
• Further validation is needed to optimize the checklist for consistent quality assessment across diverse medical modeling studies.