Problems encountered in using written criteria to assess drug information responses

Problems encountered in using written criteria to evaluate drug information responses during a study designed to determine whether simple requests were being unnecessarily submitted to a drug information center are described. Drug information questions that had been answered by a drug information center from June 1985 to June 1986 using defined references were chosen for evaluation. The center's responses to the 156 questions were paired with those of senior pharmacy students, who used the same references. Two experienced drug information pharmacists used a 6-point scale to evaluate the correctness, appropriateness, and completeness of each response, as well as the amount of extraneous material it contained. Interrater agreement was acceptable with respect to appropriateness, completeness, and amount of extraneous material; it was unacceptable with respect to correctness. When the judges re-evaluated the responses by using more explicit criteria for correctness, interrater agreement improved. The analysis excluded 32 responses that the judges found difficult to evaluate. Difficulties in assessing drug information responses arose from a lack of objective evaluation criteria, the wording of the request, the scope of the reference sources used, a lack of consistency between equivalent references, evaluator bias, and incomplete documentation. Efforts to determine whether simple drug information requests were being unnecessarily submitted to the drug information center were confounded by difficulty in using written criteria to judge the quality of responses.