A tool for the rapid postexamination analysis of examination data using classical test theory, generalizability...

Effective postexamination psychometric analysis plays an important role in supporting medical education assessments and is critical for ensuring reliability, dependability, validity, and fairness, which ultimately contribute to patient safety. However, many medical educational settings face significant barriers to achieving this, including expensive or complex software, limited psychometric expertise or programming skills, and increasing budgetary constraints regarding the need for rapid score release and quality control. This report describes a free, user-friendly tool that enables health education professions to conduct rapid and accurate postexamination analysis using classical test theory (CTT), generalizability theory (GT), and the Rasch model (RM).

The web-based tool, developed and tested in 2025 at the University of Nottingham, removes the need for complex postexamination data processing and provides real-time item performance metrics, reliability coefficients, and diagnostic reports, enabling educators to rapidly compare 3 measurement theories simultaneously for a comprehensive postexamination review.

A dataset from a high-stakes final medical assessment (489 students, 200 items) was uploaded. The tool rapidly produced CTT measures, GT variance components, and a decision study, including G-coefficient and phi-coefficient, as well as the absolute standard error of measurement and RM calibrations. Under CTT, the tool also outputs individual outcomes, histograms of marks, the true score and 68% or 95% bounds, rank scores, z scores, percentiles, and deciles. The tool also identified misfit items and persons, item-person maps, separation coefficients, and Rasch reliability. Additionally, the tool produced item characteristics curves, test characteristic curves, and test information function.

Future work includes expanding the tool by adding categorical responses to fully evaluate the quality of single-best-answer questions or multiple-choice question distractors. This aim will be achieved by analyzing plausible and implausible options and displaying the correlation between the total scores and the occurrence of each answer option for each question.