AI-assisted assessment of narrative documentation quality in ART consultations using a domain-based framework.

Area of Science:

• Reproductive Medicine
• Clinical Documentation Quality
• Artificial Intelligence in Healthcare

Background:

• High-quality clinical documentation is crucial in assisted reproductive technology (ART).
• Assessing narrative documentation quality traditionally relies on manual review, which can be subjective and time-consuming.
• Developing objective and reproducible methods for evaluating documentation is essential for quality improvement.

Purpose of the Study:

• To evaluate the quality of narrative clinical documentation in ART consultations.
• To implement and validate a transparent, reproducible, AI-assisted scoring framework for documentation assessment.
• To identify specific domains within documentation that require improvement.

Main Methods:

• A cross-sectional observational study involving 90 initial ART consultation records from 9 clinicians.
• A predefined, domain-based scoring framework assessing Regulatory Alignment, Clarity of Plan of Action, and Disclosure Readiness.
• A rule-constrained AI tool applied fixed scoring criteria, with domain scores normalized and analyzed using mixed-effects models.
• Human-AI concordance assessed via blinded review by an expert clinician; repeatability confirmed through rescoring.

Main Results:

• Significant differences in mean normalized documentation scores were observed across domains (Regulatory Alignment: 0.830, Clarity of Plan of Action: 0.677, Disclosure Readiness: 0.753).
• Clarity of Plan of Action consistently received the lowest scores across documents.
• Substantial agreement was found between AI and expert clinician scoring (weighted Kappa = 0.65), indicating high concordance.

Conclusions:

• A constrained, domain-based AI scoring framework provides transparent and reproducible quantification of narrative documentation quality in reproductive medicine.
• This AI-assisted approach can facilitate formative feedback and generate hypotheses for targeted documentation improvement.
• Future research should evaluate the impact of this AI framework on downstream clinical outcomes.