Script concordance testing in continuing professional development: local or international reference panels?

Context The PRACTICUM Institute has developed large-scale international programs of on-line continuing professional development (CPD) based on self-testing and feedback using the Practicum Script Concordance Test© (PSCT). Aims To examine the psychometric consequences of pooling the responses of panelists from different countries (composite panels) and the effect of increasing composite panel size. Method 97 cardiologists in Mexico answered a set of 62 PSCT cases/305 questions. A local panel was recruited in Mexico (n = 7). Other panelists were recruited in Argentina (n = 10) and Brazil (n = 11). Together they constituted a composite panel of 28 experts. Random panels of reference of increasing sizes (5, 10, 15, 20, and 25) were generated. Participants' scores were computed for each panel sample. Units of analysis were means of participants' scores per case. Discrimination, ranking and reliability of the scores obtained with each panel were estimated. Descriptive statistics, Pearson correlation coefficient, generalizability analysis, computation of Cronbach's alpha were used in the analyses. Results Correlation coefficients between the local and the composite panels ranged from 0.951 to 0.981. Cronbach alpha coefficient values were above 0.85 for all panels. The value of the relative G coefficient from the generalizability analyses varied from 0.91 to 0.93, indicating very high and stable ranking of participants, though absolute value of scores increased with increasing composite panel size. Conclusions In CPD environments, and with panel members selected with the highest standards, composite panels can be used. Panels of all sizes yielded high psychometric qualities. Absolute scores should be interpreted with caution.