Reliability of Large Language Model Generated Clinical Reasoning in Assisted Reproductive Technology: Blinded...

Area of Science:

• Medical Artificial Intelligence (AI)
• Reproductive Medicine
• Clinical Decision Support

Background:

• High-quality clinical chains-of-thought (CoTs) are crucial for explainable medical AI.
• Data scarcity currently limits the development of reliable CoTs.
• The clinical reliability of large language model (LLM)-generated CoTs remains unclear.

Purpose of the Study:

• To evaluate the clinical reliability of LLM-generated CoTs in reproductive medicine.
• To identify effective prompting strategies for enhancing CoT quality.

Main Methods:

• A blinded comparative study involving senior clinicians in assisted reproductive technology.
• Evaluation of CoTs generated using zero-shot, random few-shot, and selective few-shot prompting strategies.
• Comparison of expert clinician ratings with evaluations from a state-of-the-art AI model (GPT-4o).

Main Results:

• The selective few-shot strategy significantly improved CoT logical clarity, information use, and clinical accuracy (P<.001).
• Random few-shot prompting offered no significant improvement over the zero-shot baseline.
• An AI evaluator (GPT-4o) failed to distinguish critical performance differences between strategies, highlighting the need for strategic prompt design.

Conclusions:

• A "dual principles" framework combining "gold-standard depth" and "representative diversity" is proposed for generating trustworthy CoTs.
• This research addresses the data bottleneck in reproductive medicine AI development.
• Human expertise remains essential for evaluating the clinical reliability of AI-generated data.