Validation of an AI-powered mobile application for personalizing medical note explanations: a mixed-methods...

Nearly half of adults struggle to understand written health information, making medical communication a persistent barrier to effective care. While artificial intelligence has potential to improve health communication, few patient-facing tools have undergone systematic validation for personalized medical explanation.

Patiently AI is a mobile application designed to clarify clinician-authored medical notes using large language models with audience-specific adaptations (child, teenager, adult, carer) and tone variations (friendly, informative, reassuring). A three-phase mixed-methods evaluation was conducted: (1) computational readability analysis of 210 AI-generated explanations using established metrics; (2) expert review by 15 healthcare professionals assessing medical accuracy, safety, and communication quality; and (3) a patient survey of 54 participants evaluating preferences, comprehension, and acceptance.

AI-generated explanations demonstrated consistent improvements in readability, with mean Flesch-Kincaid Grade Level decreasing by 2.96 levels (10.57-7.61), Flesch Reading Ease increasing by 31.9 points (37.7-69.6), and Gunning Fog Index decreasing by 4.09 points (14.5-10.4); all improvements were statistically significant (all P ≤ 0.002). Readability gains were greatest for younger audiences (child: 4.25 grade-level reduction; adult: 1.80). Expert reviewers rated outputs highly for medical accuracy (4.49 ± 0.83/5), clarity (4.53 ± 0.77/5), and trustworthiness (4.37 ± 0.90/5), with 87.3% assessed as clinically safe. Inter-rater agreement across the 15 reviewers was substantial (Gwet's AC1 = 0.72 for safety assessments). Among patients, 70.0% of responses preferred AI-generated explanations (P < 0.001), with 98.1% comprehension accuracy and high ratings for clarity (4.58 ± 0.65/5) and confidence in care (4.19 ± 0.85/5). Overall, 70.4% indicated a likelihood of using the application.

This mixed-methods evaluation suggests that a deliberately constrained, language-focused AI system can improve the accessibility of medical notes while preserving clinical accuracy and safety. Patiently AI demonstrates a scalable approach to supporting health literacy and patient engagement without extending into clinical interpretation.