The Use of ChatGPT-4.0 to Simplify Breast Pathology Reports: A Study on Readability and Accuracy

Patients have immediate access to their diagnostic reports but these reports exceed the recommended reading level for patient-facing materials. Generative artificial intelligence may be a tool for improving patient comprehension of health information. This study assessed the readability and accuracy of ChatGPT-simplified breast pathology reports.

Ten de-identified patient breast pathology reports were simplified by ChatGPT-4.0 using three different prompts. Prompt 1 requested simplification, Prompt 2 added a 6th-grade-level specification, and Prompt 3 requested essential information. The Flesch-Kincaid Reading Level (FKRL) and Flesch Reading Ease Score (FRES) were utilized to quantify readability and ease of reading, respectively. Five physicians used a four-point scale to assess factual correctness, relevancy, and fabrications to determine overall accuracy. Mean scores and standard deviations for FKRL, FRES, and accuracy scores were compared using analysis of variance (ANOVA) and t-tests.

Prompt 2 demonstrated a reduction in FKRL (p < 0.001) and an increase in FRES (p < 0.001), demonstrating improved readability and ease of reading. ChatGPT-simplified reports received an overall accuracy score of 3.59/4 (standard deviation [SD] ± 0.17). The scores by rubric category were 3.62 (SD ± 0.31) for factual correctness (4 = completely correct), 3.27 (SD ± 0.44) for relevancy (4 = completely relevant), and 3.89 (SD ± 0.11) for fabricated information (4 = no fabricated information).

ChatGPT simplified breast pathology reports to the reading level recommended for patient-facing materials when given a grade-level specification while mostly maintaining accuracy. To minimize the risk of medically inaccurate and/or misleading information, ChatGPT-simplified reports should be reviewed before dissemination.