A new instrument to measure appropriateness of services in primary care

To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits.

Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits.

Primary care practices in a staff model health maintenance organization and a large preferred provider network.

Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic.

For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3% of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56% for new medication, 31% for diagnostic test, and 22% for referral.

This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.