Real-time analysis of Delphi panel data to facilitate iterative rating within meetings

The Modified Delphi panel process involves iterative ratings, often with lengthy lags between rounds to allow time for data analysis. We sought to develop, implement, and evaluate an approach for real-time data analysis and presentation to complete multiple rounds of ratings in a single panel meeting.

We developed statistical code in Python to analyze Delphi panel ratings completed in Slido using a 9-point Likert scale. The code generates histograms of panel ratings. The median score is color-coded pink when formal disagreement criteria are met (regardless of median score), and when there is no disagreement, red for median scores of 1-3, yellow for 4-6, and green for 7-9. We implemented this approach for a Delphi panel on sepsis recovery that rated three sets of care practices. Round 1 rating, panel discussion, and round 2 rating all occurred in one day for each set of practices. Round 1 ratings were presented during discussion. This approach avoided "homework" outside meetings. All panelists either strongly agreed (57%) or agreed (43%) it was easy to understand the presentation of round 1 ratings. Most panelists either strongly agreed (50%) or agreed (21%) that they liked doing both rounds of rating during the meetings.