Feasibility of Privacy-preserving Locally Hosted Language Models for Assessing Medical Student Reflective Writing

Reflective writing assessment in medical education is labor-intensive and raises data privacy concerns that preclude cloud-based AI solutions. We conducted a feasibility study using a locally-hosted, open-weight large language model (LLM) deployed on institutional servers to assess medical students' reflective writing. The model evaluated 10 de-identified student narratives using two published rubrics across three independent passes, generating rubric-aligned scores and formative feedback. Scores were consistent across passes and rubrics with no statistically significant differences. Expert faculty rated LLM-generated feedback as meeting most quality criteria (mean 5.48/7). These findings support feasibility of a privacy-preserving, locally-hosted LLM for scalable reflective writing assessment.

The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02865-7.