Is the evaluation of the personal statement a reliable component of the general surgery residency application?

Each year, fourth-year medical students spend considerable time writing and rewriting their personal statements. However, there is little evidence of what role the personal statement plays in deciding which applicants will be invited for an interview.

To evaluate the inter-rater reliability of a surgical selection committee's ratings for both the personal statement and the application summary parts of the residency application.

We completed a retrospective analysis of the 2007-2008 Scott & White surgical residency application pool. From a total pool of 174 residency applications, we selected 8 (5%) applications randomly to be evaluated by 4 experienced members of the selection committee. The 4 committee members rated each personal statement on a 7-point scale, from "negative-would not invite for an interview" to "positive-will invite for an interview." They rated respective application summaries separately on a similar 7-point scale. Committee members also listed their top three reasons for assigning their scores.

Rating scores for the personal statements and the applications were analyzed for inter-rater correlation. The qualitative data (ie, reasons for the scores) were reviewed to help the investigators profile the reasons given for very positive and very negative scores.

For the application summaries, the correlations between each pair of raters ranged from 0.79 to 0.94 with an overall average of 0.88. For the personal statements, inter-rater correlations ranged from -0.83 to 0.63 with an overall average of -0.09.

These results demonstrate that the personal statements lacked objective criteria for evaluation.