Human Consensus Versus Generative Artificial Intelligence When Assessing the Methodological Quality and Overall...

To evaluate the level of agreement between ChatGPT-5.5 and human assessments when appraising the methodological quality and overall confidence of the results of systematic reviews (SRs) with meta-analysis (MA) in endodontology.

SRs with MA related to endodontology and published in English between 2021 and 2024 were included. The methodological quality of the SRs was appraised by three human experts using AMSTAR 2, and a consensus dataset was generated. ChatGPT-5.5 was applied to generate AMSTAR 2 item-level ratings and overall confidence classifications for each SR through three newly created accounts. Agreement between consolidated ChatGPT-5.5 ratings and human consensus was assessed for both overall confidence and each AMSTAR 2 item. Overall-confidence agreement was evaluated under the primary criterion (Partial Yes = Yes) and an alternative sensitivity criterion (Partial Yes = No). Direct agreement and linearly weighted Cohen's kappa were calculated, and complete 3/3 agreement was used to assess consistency across ChatGPT-5.5 accounts.

One hundred SRs were included. Human consensus and ChatGPT-5.5 classified 47 and 85 SRs, respectively, as critically low. Under the primary criterion (Partial Yes = Yes), overall confidence matched in 52/100 SRs (52%; weighted κ = 0.182). Under the alternative criterion (Partial Yes = No), agreement increased to 83%, while weighted κ remained similar (κ = 0.173), indicating that direct agreement was sensitive to the handling of Partial Yes. Item-level agreement varied substantially. Overall confidence was identical across the three ChatGPT-5.5 accounts in 85 SRs, and global item-level 3/3 agreement was 86.12%. In the 100 sampled SRs, 15% met the criterion for reporting primary-study funding and 21% for justifying study designs.

Overall-confidence direct agreement was sensitive to the handling of Partial Yes ratings, whereas chance-corrected agreement remained similar under both criteria. ChatGPT-5.5 assigned lower overall confidence classifications more frequently than human consensus, and agreement varied across AMSTAR 2 domains. Although outputs were frequently consistent across the three ChatGPT-5.5 accounts, between-account consistency did not ensure concordance with human judgements. The findings do not support the use of ChatGPT-5.5 as an independent AMSTAR 2 assessor; any exploratory application requires item-by-item verification by reviewers with methodological expertise.