Rating versus ranking in a Delphi survey: a randomized controlled trial

Area of Science:

• Health Services Research
• Decision Science

Background:

• The Delphi technique is increasingly used in health research for structured group communication.
• Rating and ranking are common methods to quantify participant opinions in Delphi surveys.
• This study investigated the impact of rating versus ranking on item prioritization, completion time, and task difficulty in primary cardiovascular care organization.

Purpose of the Study:

• To compare the influence of rating versus ranking approaches on item prioritization in a Delphi survey.
• To assess the impact of rating versus ranking on questionnaire completion time.
• To evaluate participant perception of task difficulty between rating and ranking methods.

Main Methods:

• A randomized controlled trial embedded within a three-round online Delphi survey.
• Participants (patients, patient partners, clinicians) were allocated to either a rating or ranking group.
• Item prioritization agreement was measured using Krippendorff's alpha; task difficulty was assessed via the Single Ease Question.

Main Results:

• Both rating and ranking groups identified similar top priorities, with discrepancies emerging by the seventh item.
• Moderately strong agreement (Krippendorff's alpha = 0.811) was observed for the top items between groups.
• Ranking was perceived as significantly more difficult than rating (p < 0.001), with no significant difference in completion time.

Conclusions:

• Rating and ranking procedures result in modestly similar item prioritization in Delphi surveys.
• Ranking is associated with greater perceived task difficulty compared to rating.
• Replication with larger sample sizes and varied procedures is suggested.