The reliability of radiologists' quality assessment of MR pelvic scans

The purpose of this study was to test the reliability of radiologists' subjective assessment of the quality of MR scans of the pelvis. A four-grade ordinal subjective scale, based on the degree of artifact and contrast between pelvic organs, was developed in a pilot study by two MR radiologists. Forty pelvic scans were graded "blindly" in random order by the same two "practiced" MR radiologists, and an objective measurement of scan artifact was obtained for each scan. Twenty-eight pelvic scans were also graded by two "unpracticed" radiologists not involved in the development of the scale. For the practiced radiologists, the interobserver percentage agreement was 80% (weighted kappa of 0.78) and the intraobserver percentage agreement was 75% (weighted kappa of 0.73). For the unpracticed radiologists the percentage agreement was 61% and the weighted kappa was 0.55. The correlation between the subjective and objective measurement was only 0.27. In conclusion, the objective measurement of scan artifact showed poor correlation with the radiologists' subjective assessment of scan quality. The subjective assessment demonstrated satisfactory reliability and, therefore, could be considered as an additional outcome measurement for scan quality in clinical trials or as a relevant measure of quality assurance.