Prompt Framing and Evidence Requirements for AI-Generated Educational Responses in Dental Education: An Experimental...

Large language models (LLMs) are increasingly used in health professions education; however, the role of prompt design in shaping their outputs remains poorly understood in clinical training contexts. In dentistry, where the presentation of information, perceived credibility, and procedural reasoning are important, the effects of instructional framing and evidence requirements on artificial intelligence (AI)-generated educational responses are particularly relevant.

This study examined whether instructional framing and evidence requirements were associated with rater-assessed perceived factuality, confidence of tone, stance orientation, citation behavior, safety notices, hedging, and response length in cavity preparation responses generated using the web-based ChatGPT-5.

In a 2 × 2 factorial experiment, we manipulated instructional framing (patient-centered vs. skill-centered) and evidence requirement (evidence-required vs. no evidence) across 10 base prompt topics and four experimental conditions, yielding 40 outputs. Five trained raters independently coded each response for perceived factuality, confidence of tone, stance orientation, hedging, citation presence, safety notices, and response length. Inter-rater reliability was assessed using intra-class correlation coefficients and Fleiss' κ. Consensus scores were analyzed using factorial analyses of variance and chi-square tests.

Requiring evidence was associated with greater citation presence (19/20, 95%, vs. 5/20, 25%; P<.001) and longer responses (P<.001). In exploratory analyses, it was also associated with higher ratings of perceived factuality (P=.015). Hedging and confidence of tone showed non-significant patterns. Instructional framing was associated with stance orientation (P=.006) but was not associated with response length. No refusals occurred, and safety notices were infrequent across conditions. Inter-rater reliability was high for citation presence but low or variable for several subjective measures, particularly perceived factuality and safety notices.

Prompt design was associated with differences in the presentation, structure, and stance orientation of LLM-generated educational responses in dentistry. Evidence requirements increased citation inclusion and response length, whereas instructional framing was associated with the stance emphasized in the response. These findings suggest that deliberate, pedagogically aligned prompt design may support the development and evaluation of AI-generated content in dental education. However, the effects of prompt wording on objectively verified accuracy, clinical safety, and learning outcomes require further investigation.