Quantifying Radiology Residents' Learning Curves in Report Writing Performance Through Report Comparison and Jaccard...

Background Report writing skills are a core competency to be acquired during residency, yet objective tools for tracking performance are lacking. Purpose To investigate whether the Jaccard index, derived from report comparison, can objectively illustrate learning curves in report writing performance throughout radiology residency. Materials and Methods Retrospective data from 246 984 radiology reports written from September 2017 to November 2022 in a tertiary care radiology department were included. Reports were scored using the Jaccard similarity coefficient (ie, a quantitative expression of the amount of edits performed; range, 0-1) of residents' draft (unsupervised initial attempt at a complete report) or preliminary reports (following joint readout with attending physicians) and faculty-reviewed final reports. Weighted mean Jaccard similarity was compared between years of experience using Welch analysis of variance with post hoc testing overall, per imaging division, and per modality. Relationships with years and quarters of resident experience were assessed using Spearman correlation. Results This study included 53 residents (mean report count, 4660 ± 3546; 1-5 years of experience). Mean Jaccard similarity of preliminary reports increased by 6% from 1st-year to 5th-year residents (0.86 ± 0.22 to 0.92 ± 0.15; P < .001). Spearman correlation demonstrated a strong relationship between residents' experience and higher report similarity when aggregated for years (r_s = 0.99 [95% CI: 0.85, 1.00]; P < .001) or quarters of experience (r_s = 0.90 [95% CI: 0.73, 0.96]; P < .001). For residents' draft reports, Jaccard similarity increased by 14% over the course of the 5-year residency program (0.68 ± 0.27 to 0.82 ± 0.23; P < .001). Subgroup analysis confirmed similar trends for all imaging divisions and modalities (eg, in musculoskeletal imaging, from 0.77 ± 0.31 to 0.91 ± 0.16 [P < .001]; r_s = 0.98 [95% CI: 0.72, 1.00] [P < .001]). Conclusion Residents' report writing performance increases with experience. Trends can be quantified with the Jaccard index, with a 6% improvement from 1st- to 5th-year residents, indicating its effectiveness as a tool for evaluating training progress and guiding education over the course of residency. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Bruno in this issue.