Related Experiment Video
Updated: May 24, 2026

Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Reliability and credibility of progress test criteria developed by alumni, faculty, and mixed alumni-faculty judge
H Glenn Anderson1, Arthur A Nelson
1Texas Tech University Health Sciences Center, School of Pharmacy, USA. glenn.anderson@marshall.edu
Objective:
To compare the reliability and credibility of Angoff-based, absolute criteria derived by faculty, alumni, and a combination of alumni and faculty judge panels.
Methods:
Independently, faculty, alumni, and mixed faculty-alumni judge panels developed pass/fail criteria for an 86-item test. Generalizability and decision studies were performed. Root mean square errors (RMSE) and 95% confidence intervals were calculated for reliability and credibility assessment. School graduate performance upon the North American Licensure Examination (NAPLEX) was the comparator for credibility assessment.
Results:
RMSEs were 1.06%, 1.42%, and 2.32% for the alumni, faculty, and mixed judge panels respectively. The school's NAPLEX pass rate was 97.5%. This rate triangulated well with the faculty judge panel (pass rate = 93.9%, CI95% = 87.1%-98.2%), but did not with either mixed judge or alumni judge panels.
Conclusions:
Faculty-derived criteria offer superior pass/fail decision defensibility relative to both alumni derived and mixed faculty-alumni derived criteria.
Related Concept Videos
Reliability and Validity
Surveys
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...
Kendall's Coefficient of Concordance
Significance Testing: Overview
Self-Evaluation: Self-Enhancement and Self-Verification