Related Experiment Videos
Readability and Content Integrity of Discharge Documents for Common Pediatric Orthopaedic Procedures Before and After
Angel G A Prempeh1, Rashid Saeed, Christina Hill
1Michigan State University College of Human Medicine, East Lansing, MI.
Background:
Caregivers of pediatric orthopaedic surgical patients routinely receive discharge documents written at reading levels exceeding established health literacy recommendations. This disparity creates a significant barrier to comprehension and adherence to postoperative instructions. While artificial intelligence (AI) language models offer a scalable solution for simplification of medical text, a systematic evaluation of the ability to preserve critical content in surgical discharge materials remains lacking.
Methods:
Publicly available caregiver-facing discharge documents for 3 common pediatric orthopaedic conditions, supracondylar humerus fractures (CRPP), pediatric forearm fractures (ORIF/CRPP), and slipped capital femoral epiphysis (SCFE pinning), were collected. All documents underwent AI simplification using a standardized prompt with ChatGPT (version 5.2; OpenAI). Readability was assessed using 6 validated indices for both original and simplified documents. A structured content integrity review was conducted on all simplified documents by 2 independent reviewers to assess for safety-critical errors, completeness gaps, and factual accuracy, with inter-rater reliability quantified by Cohen kappa.
Results:
Fifteen documents were obtained from institutions across 9 US states. After AI simplification, mean Flesch-Kincaid Grade Level (FKGL) decreased from 8.91 (SD: 1.38) to 4.97 (SD: 2.14), a mean reduction of 3.94 grade levels (95% CI: 2.65-5.23; P<0.001; Cohen d=1.69). All 6 readability indices improved across all documents. Eleven of 15 documents (73%) achieved the target readability threshold of FKGL 6 or below, compared with zero at baseline. Content integrity review identified 35 flags across 14 of 15 documents: 12 safety-critical errors, 16 completeness gaps, and 7 accuracy concerns. Cohen kappa was 0.87 for agreement on whether a change was clinically meaningful, 0.85 for severity classification, and 0.89 for flag category. Three documents had content omissions sufficient to require clinician revision before clinical use. Greater word-count reduction was associated with a higher burden of safety-critical flags (Spearman rho 0.63; P=0.012).
Conclusions:
While ChatGPT substantially reduced the reading complexity of pediatric orthopaedic discharge documents, it also introduced safety-critical alterations. AI-assisted simplification should be implemented only with structured clinician review. Larger word-count reductions were associated with more safety-critical alterations, so the magnitude of reduction may help prioritize which simplified documents receive the closest clinician review.
Level Of Evidence:
Level III.
