Related Experiment Videos
Setting standards for effectiveness: a comparison of expert panels and decision analysis
S J Bernstein1, T P Hofer, A P Meijler
1Department of Medicine, University of Michigan, Ann Arbor 48109-0376, USA.
Insights
Dutch physicians showed substantial agreement with decision analysis for coronary revascularization procedures, unlike U.S. physicians. This highlights variations in expert panel consensus compared to decision analytic models.
Area of Science:
- Cardiology
- Health Services Research
- Medical Decision Making
Background:
- Coronary revascularization procedures, including coronary artery bypass graft surgery (CABG) and percutaneous transluminal coronary angioplasty (PTCA), are common treatments for chronic stable angina.
- Establishing appropriate criteria for these interventions is crucial for optimal patient outcomes and resource allocation.
Purpose of the Study:
- To compare the appropriateness criteria for coronary revascularization developed by expert physician panels with those derived from decision analysis.
- To assess the agreement between U.S. and Dutch expert panels and between each panel and a decision analytic model.
Main Methods:
- A review of medical records for 3080 patients with chronic stable angina who underwent CABG or PTCA.
- Two expert physician panels (U.S. and Dutch) rated the appropriateness of revascularization procedures.
- Agreement was assessed between panels and with a decision analytic model's effectiveness categories using kappa statistics.
Main Results:
- Poor agreement was found between U.S. and Dutch panels for PTCA (kappa = 0.03) and slight agreement for CABG (kappa = 0.18).
- Dutch panel ratings showed substantial agreement with the decision analytic model for both PTCA (kappa = 0.83) and CABG (kappa = 0.79).
- U.S. panel ratings showed no systematic agreement for PTCA (kappa = 0.00) and poor agreement for CABG (kappa = 0.18) with the decision analytic model.
Conclusions:
- Agreement between expert panels and decision analysis varies by procedure and evidence strength.
- Dutch physicians demonstrated significantly higher agreement with decision analysis compared to U.S. physicians regarding coronary revascularization appropriateness.
- These findings suggest potential differences in the interpretation or application of evidence-based guidelines between international expert groups.
Objective:
To compare criteria for coronary revascularization developed by the expert panel process and by decision analysis.
Method:
We reviewed the medical records of 3080 chronic stable angina patients who either underwent coronary artery bypass graft surgery (CABG) or percutaneous transluminal coronary angioplasty (PTCA) and determined the agreement between appropriateness ratings made by two expert physician panels, one from the United States and the second from The Netherlands. We also evaluated the agreement between these panels' appropriateness ratings and a decision analytic model's effectiveness categories.
Results:
There was poor agreement between U.S. and Dutch panel appropriateness ratings for PTCA (kappa = 0.03) and slight agreement for bypass surgery (kappa = 0.18). Dutch ratings had substantial agreement with the decision analytic models effectiveness categories for both PTCA and CABG (kappa = 0.83 and 0.79, respectively) whereas there was no systematic agreement between U.S. ratings and the decision analytic model for PTCA and poor agreement for CABG (kappa = 0.00 and 0.18, respectively).
Conclusions:
Although the level of agreement between expert panels and decision analysis on when a procedure is appropriate or effective may vary by procedure and the strength of the scientific evidence, we found that Dutch physicians agree much more strongly with decision analysis than U.S. physicians.