The Use of Automated Linguistic Analyses to Support Experiential Quality Assurance

The objective was to combine a set of linguistic analyses for efficiently analyzing large amounts of qualitative data from student evaluations of experiential sites and preceptors for quality assurance, evaluate their accuracy in coding these evaluations compared with manual processes, and assess the correlation between quantitative and qualitative sections of student evaluations.

A Python script was written to analyze comments from the qualitative sections of deidentified student evaluations of their preceptor and site. Each comment was analyzed using lexical, sentiment, and semantic analyses and given an aggregate rating, ranging from very negative to very positive. The script was iteratively refined using randomly selected samples of evaluations, subsequently tested to verify accuracy, and then used to analyze a cohort of evaluations. The correlation between the quantitative and qualitative sections was assessed to determine the strength of the relationship between the 2 sections.

After refinements to the script, 93% of the qualitative sections of evaluations showed the correct valence (eg, positive, neutral, negative), as measured against agreed-upon experiential faculty ratings. Of the 7% coded incorrectly, 2% were programmatically relevant. The correlation between quantitative and qualitative sections was weak but significant.

The linguistic analyses were able to identify sites potentially requiring further review, holding promise for alleviating burden on experiential staff by limiting the quantity of qualitative data to manually review. Correlation results suggested that quantitative data were not sufficient to capture potential concerns. Further research is necessary to evaluate integration into real-time experiential education workflows.