Evaluating judge performance in sport

Many sports, such as, gymnastics, diving, ski jumping, and figure skating, use judges' scores to determine the winner of a competition. These judges use some type of rating scale when judging performances (e.g., figure skating: 0.0 - 6.0). Sport governing bodies have the responsibility of setting and enforcing quality control parameters for judge performance. Given the judging scandals in figure skating at the 1998 and 2002 Olympics, judge performance in sport is receiving greater scrutiny. The purpose of this article is to illustrate how results from Rasch analyses can be used to provide in-depth feedback to judges about their scoring patterns. Nine judges' scores for 20 pairs of figure skaters who competed at the 2002 Winter Olympics were analyzed using a four-faceted (skater pair ability, skating aspect difficulty, program difficulty, and judge severity) Rasch rating scale model that was not common to all judges. Fit statistics, the logical ordering of skating aspects, skating programs, and separation indices all indicated a good fit of the data to the model. The type of feedback that can be given to judges about their scoring pattern was illustrated for one judge (USA) whose performance was flagged as being unpredictable. Feedback included a detailed description of how the rating scale was used; for example, 10% of all marks given by the American judge were unexpected by the model (Z > |2|). Three figures illustrated differences between the judge's observed and expected marks arranged according to the pairs' skating order and final placement in the competition. Scores which may represent "nationalistic bias" or a skating order influence were flagged by looking at these figures. If sport governing bodies wish to improve the performance of their judges, they need to employ methods that monitor the internal consistency of each judge as a many-facet Rasch analysis does.