Simultaneous confidence intervals for ranks with application to ranking institutions

Area of Science:

• Statistics
• Data Analysis
• Ranking Methodologies

Background:

• Institutional rankings often rely on performance measures with inherent uncertainty.
• Accurate assessment of rank uncertainty is crucial for decision-making in fields like healthcare and higher education.
• Existing methods for rank uncertainty may lack precision or computational efficiency.

Purpose of the Study:

• To develop a novel statistical method for constructing simultaneous confidence intervals (CIs) for true institutional ranks.
• To address the limitations of existing methods in accurately reflecting rank uncertainty.
• To provide a more reliable approach for comparing institutions based on performance data.

Main Methods:

• A new method based on Tukey's honest significant difference test for simultaneous CIs of true ranks.
• Development of a rescaling technique to adjust for conservatism in non-tied performance data.
• Comparison with a Monte-Carlo based method and a method using simultaneous CIs for performance measures.

Main Results:

• The proposed method achieves nominal coverage levels when true performances are equal and is conservative otherwise.
• A rescaling method improves CI length while maintaining coverage control for non-tied data.
• The novel method provides uniformly shorter CIs compared to a similar existing approach.
• The Monte-Carlo method, when rescaled, performs similarly but is computationally intensive.

Conclusions:

• The novel method offers a statistically sound and computationally efficient approach to assessing rank uncertainty.
• Rescaling improves the performance of both the proposed and Monte-Carlo methods.
• The developed technique provides superior CIs for institutional rankings, demonstrated with real-world data examples.