Related Experiment Videos
The reliability of peer assessments. A meta-analysis
1VA Central Office, Washington, DC 20420.
Evaluation & the Health Professions
|February 7, 1994
Summary
Peer assessments of care quality show limited reliability, with a weighted mean kappa of .31. Improvements are needed in peer review processes or quality measurement methods for better healthcare quality assessment.
Area of Science:
- Healthcare quality assessment
- Medical peer review
- Health services research
Background:
- Peer assessment is a standard method for evaluating healthcare quality of care.
- Assessing the reliability of peer review is crucial for its validity.
Purpose of the Study:
- To conduct a meta-analysis of studies on the interrater reliability of peer assessments of quality of care.
- To evaluate the consistency of judgments made by different peer reviewers.
Main Methods:
- A systematic literature search was performed using Medline, Health Planning and Administration, and SCISEARCH databases.
- Included English-language studies published from 1966 to 1991 that reported chance-corrected agreement (kappa) among peer reviewers.
- A weighted mean kappa was calculated from 21 independent findings across 13 studies.
Main Results:
- The meta-analysis yielded a weighted mean kappa of .31.
- This kappa value indicates limited interrater reliability for standard peer assessments of quality of care.
- The findings suggest significant variability in how peer reviewers evaluate care.
Conclusions:
- The interrater reliability of peer assessment in healthcare quality evaluation is currently quite limited.
- Further research is essential to enhance the reliability of the peer review process.
- Alternative or improved quality indicators with established validity and reliability should be explored.