Use of controlled vignettes in evaluation: does type of response method make a difference?

Two studies were performed using vignettes as a method in program evaluation. One study used a rating scale method. The second study used a forced choice decision method. The two studies were performed with students from the pre-Occupational Therapy (pre-OT) program, the Occupational Therapy (professional OT) program and the College of Business at Colorado State University. Students were asked to respond to a questionnaire about how a hypothetical job applicant with a cognitive disability would perform in one of three types of job settings: a segregated, sheltered workshop setting; an integrated supported employment setting; or an integrated competitive employment setting. The students in the Professional Occupational Therapy program had been part of a training grant, 'An integrated approach to personnel preparation for transition', which focused on the integration of transition competencies into an occupational therapy curriculum. The forced choice vignette method resulted in statistically significant positive outcomes toward supported employment and the influence of the training grant. The discussion focused on the different outcomes of the two studies relative to the methods applied and the use of vignettes as an evaluation tool to assess the impact of a modified curriculum. Attention was directed toward the problem of external validity when using vignettes (Gorman, C.D., Clover, W.H. & Doherty, M.E., (l978). Can we learn anything about interviewing real people from 'interviews' of paper people? Two studies of the external validity of a paradigm. Organizational Behavior and Human Performance, 22, 165-192.).