Evaluating diagnostic pattern recognition: the psychometric characteristics of a new item format

The issues related to measuring pattern recognition are similar to measuring other clinical skills; how can testing time be used most efficiently to obtain reliable and valid scores? How should tests be constructed to obtain scores that validly reflect individual performance in making diagnoses? Generalizability analyses indicated that performance in one topic area does not predict performance in other areas very well. For example, students who were relatively expert in diagnosing patients with headaches tend not to be expert in diagnosing patients with chest pain, joint pain, etc. Therefore, to evaluate diagnostic pattern recognition skills in general, it is preferable to sample more presenting complaints with fewer items directed at each one, rather than to sample more items within a small number of presenting complaints. Approximately one hour of testing time would be required to generate a reasonably reliable score (ie, with a generalizability coefficient greater than 0.80). For diagnostic or remedial purposes, performance can be examined by content area to determine specific areas of weakness for individual students. The next phase of this study will be directed at determining whether there are benefits to using the current matching format with a relatively long list of response alternatives rather than a traditional multiple choice format with five choices. It is hypothesized that the shorter list differentially benefits the lower ability students and the more junior students. Efforts will also be directed to determining the applicability of the item format to other content areas such as diagnostic testing and therapy.