The reliability and validity of in-training evaluation reports in obstetrics and gynecology

In a specific Obstetrics and Gynecological program, the program and certifying ITERs were evaluated for their measurement qualities. The internal consistency of the ITERs is supported. The tendency for high inter-item correlations suggest overall judgment of candidates may be influencing individual item rankings--particularly on the Program ITER. Unfamiliarity of faculty with appropriate behaviors may be one of the reasons for this effect based on the faculty's inability to select correct behavior for each item. Very limited inter-form consistency is noted and random associations of items often correlate higher than parallel items. The stability of the Program ITER is supported, but there is little support for criterion validity based on the criterion variables available. It is concluded that more clearly defined behaviors must be identified for each ITER item and faculty must be trained in their use. The use of the same ITER for all specialities may be a major reason for this inconsistency. Lastly, more studies of validity are advised.