Forced choice and ordinal discrete rating assessment of image quality: a comparison

This study compared a five-category ordinal scale and a two-alternative forced-choice subjective rating of image quality preferences in a multiabnormality environment. 140 pairs of laser-printed posteroanterior (PA) chest images were evaluated twice by three radiologists who were asked to select during a side-by-side review which image in each pair was the "better" one for the determination of the presence or absence of specific abnormalities. Each pair included one image (the digitized film at 100 microns pixel resolution and laser printed onto film) and a highly compressed (approximately 60:1) and decompressed version of the digitized film that was laser printed onto film. Ratings were performed once with a five-category ordinal scale and once with a two-alternative forced-choice scale. The selection process was significantly affected by the rating scale used. The "comparable" or "equivalent for diagnosis "category was used in 88.5% of the ratings with the ordinal scale. When using the two-alternative forced-choice approach, noncompressed images were selected 66.8% of the time as being the "better" images. This resulted in a significantly lower ability to detect small differences in perceived image quality between the noncompressed and compressed images when the ordinal rating scale is used. Observer behavior can be affected by the type of question asked and the rating scale used. Observers are highly sensitive to small differences in image presentation during a side-by-side review.