Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc

Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either "include," "uncertain," or "exclude." There was moderate to substantial agreement between raters for "include" and "exclude," but little to no agreement for "uncertain." In most cases only a single rater used the "uncertain" classification for a given participant's data, with the remaining raters showing agreement for "include"/"exclude" decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for "uncertain" cases, aiding classification consistency.