Identifying Diagnostic Breakdowns in Patient Experience Survey Comments Using Natural Language Processing: An Evaluation Study

Patient experience surveys may contain signals of diagnostic breakdowns that are difficult to identify through conventional safety surveillance. Natural language processing (NLP) may enable these narratives to be analyzed at scale.

To develop and evaluate three increasingly context-sensitive NLP approaches for identifying patient-perceived diagnostic breakdowns in ambulatory patient experience survey comments and to explore the potential of prompt-engineered large language models (LLMs) for this task.

We analyzed 523 free-text comments from a 2023 ambulatory patient experience survey at a Canadian academic community hospital. Comments were manually coded using a structured framework to establish a finalized reference standard. Three sequential approaches were developed: a TF-IDF-based XGBoost classifier (Phase I), a fine-tuned Twitter-RoBERTa model (Phase II), and a locally deployed LLaMA 3.3 70B model refined through iterative expert-guided prompt development (Phase III). Phases I and II were evaluated using 5×5 nested stratified cross-validation. Phase III was refined and evaluated on the full dataset and therefore represents development-set rather than independent held-out performance.

Following expert review, 51 of 523 comments (9.8%) were classified as diagnostic breakdowns. Compared with Phase I, Phase II showed higher recall (0.767 vs 0.626), F1 score (0.620 vs 0.548), AUROC (0.921 vs 0.810), and AUPRC (0.564 vs 0.490). In Phase III, iterative prompt refinement reduced over-classification of general experiential complaints. The final prompt achieved recall of 0.961, precision of 0.803, specificity of 0.975, and an F1 score of 0.875 on the development dataset.

Patient-authored survey narratives are an underused source of information for diagnostic safety and quality improvement. Context-sensitive NLP approaches may improve detection of subtle diagnostic concerns, and the strong development-set performance of the prompt-engineered LLM supports further study. External validation and prospective evaluation within diagnostic-safety workflows are needed before broader implementation.