AI-assisted assessment of narrative documentation quality in ART consultations using a domain-based framework

To assess narrative clinical documentation quality in assisted reproductive technology (ART) consultations, by applying a transparent, reproducible AI-assisted scoring framework.

In this cross-sectional observational study, 90 consecutive initial ART consultation records from 9 clinicians were evaluated using a predefined, domain-based documentation scoring framework. Documentation quality was assessed across three domains: Regulatory Alignment, Clarity of Plan of Action, and Disclosure Readiness. A rule-constrained AI-assisted tool applied fixed scoring criteria. Domain scores were normalised, and mixed-effects models were used to examine domain differences, associations, and clinician-level effects. Human-AI concordance was assessed in a blinded random subset scored by an independent expert clinician. Repeatability was assessed through repeated rescoring under identical conditions.

Mean normalised documentation scores differed significantly across domains (Regulatory Alignment 0.830, Clarity of Plan of Action 0.677, Disclosure Readiness 0.753, Wald χ² = 284.43 p < 0.001). Domain ordering was consistent across documents, with lowest scoring for Plan Clarity (Kendall's W = 0.53, p < 0.001), and was unchanged after adjustment for consultation type and documentation length. Score agreement between AI and the expert was substantial (weighted Kappa = 0.65), with high concordance in domain ordering (mean Kendall T = 0.84) and no significant differences overall (p = 0.10).

A constrained, domain-based AI scoring framework enables transparent and reproducible quantification of narrative documentation quality in reproductive medicine. This approach may help generate formative feedback and hypotheses for targeted documentation improvement, which should be evaluated against downstream outcomes in future work.