Inter-rater reliability and usability of CATHIS core for homeopathic intervention studies

The Critical Appraisal Tool for Homeopathic Intervention Studies (CATHIS) core is a streamlined appraisal tool for homeopathic intervention studies focusing on credibility, coherence, and clinical relevance. The aim of the research project was to evaluate its inter-rater reliability, feasibility, and face validity.

In a preregistered cross-sectional study, four raters independently applied CATHIS core to 28 trials (21 randomised controlled trials, 7 non-randomised studies on interventions) drawn from reviews on insomnia and hypertension; two external reviewers provided consensus ratings. Inter-rater reliability (IRR) was estimated using percent agreement, Fleiss' κ, and Gwet's AC2 (95% CIs). Feasibility was quantified as rating time and consensus time. Associations among the three domains were explored with correlation analyses and sensitivity checks.

IRR varied markedly by domain. Credibility showed good agreement (Fleiss' κ=0.66, 95% CI 0.57-0.74; AC2 =0.76, 0.71-0.82). Coherence yielded only poor-to-fair agreement (κ=0.28, 0.16-0.40; AC2 =0.41, 0.30-0.51). Clinical relevance was similarly limited (κ=0.32, 0.23-0.41; AC2 =0.36, 0.28-0.44). Individual ratings required on average 65.8 min, while consensus discussions averaged 17.7 min. Correlation analyses indicated heterogeneous and partly overlapping domain signals with limited interpretability. Face-validity responses reflected moderate-to-high acceptance but difficulties in consistent application.

CATHIS core yielded reproducible credibility ratings but only fair and operationally fragile agreement for coherence and clinical relevance, alongside non-trivial rating burden. Taken together, the current reliability profile is insufficient for confident use in systematic reviews. Targeted refinement appears warranted before broader implementation.