Development of a radiology faculty appraisal instrument by using critical incident interviewing

To develop a valid and reliable radiology faculty appraisal instrument based on scientific methods.

Fifteen radiology residents participated in critical incident interviewing. During a 1-hour interview, a resident was asked to describe five incidents each of effective and ineffective faculty behavior. Two investigators independently listened to the tape-recorded interviews, and two different investigators sorted the incidents into broad categories. A faculty appraisal instrument was developed by listing similar incidents under broad categories. A five-point rating scale was applied to each item. Content validity was assessed by resident and faculty critique of the appraisal instrument.

A total of 168 incidents of faculty behavior were generated. The frequency with which similar incidents were reported was recorded. The most common behaviors reported were related to staff expertise and teaching. Interjudge reliability was good, as determined by computing K indices of agreement (overall K = 0.59). There was good agreement regarding instrument content validity among residents but not among faculty.

Residents supported the use of the new appraisal instrument, but further tests of validity and reliability and faculty acceptance of the instrument will determine its usefulness as a tool for monitoring faculty teaching performance and making decisions regarding faculty promotion.