National rankings as a means of evaluating medical school library programs: a comparative study

A comparative study was undertaken to assess the reasons for the low rankings received by George Washington University Medical Center library in the Annual Statistics for Medical School Libraries in the United States and Canada. Although internal studies showed the library was successfully satisfying user needs and meeting its primary objectives, the rankings, which include the traditional measures of quality used by accrediting bodies, indicated the contrary. Three hypotheses were postulated to account for the discrepancy. In a matched group of similar libraries: (1) the rankings of an individual library would differ from the national rankings; (2) clustering the variables would change the rankings; and (3) libraries with similar staff size would tend to rank in the same quartile in service and resource variables. All hypotheses were invalidated. Further tests led to the conclusion that the Annual Statistics and other traditional measures of quality are inappropriate and inaccurate methods for evaluating library programs, since they only measure resource allocations and not the effectiveness of those allocations. Alternative evaluation methods are suggested.