Large Language Models in Professionalism and Ethical Decision-Making Education: A Transferable Process for Generating...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Professional Ethics

Background:

• Concordance of Professional Judgment (CoPJ) tools aid ethical decision-making practice but are resource-intensive to develop.
• Existing CoPJ item development processes limit curriculum integration and learner practice.
• Large language models (LLMs) offer a potential solution to scale CoPJ item generation.

Purpose of the Study:

• To implement and evaluate a transferable process for generating CoPJ items using general-purpose LLMs.
• To assess the suitability of LLM-generated CoPJ items against established ethical frameworks.

Main Methods:

• Utilized a zero-shot prompt to generate CoPJ items for 28 topics using ChatGPT-5 Thinking and Gemini 2.5 Pro.
• Faculty reviewers independently rated 56 LLM-generated items using a 20-criterion checklist and global suitability judgment.
• Assessed items based on acceptance, minor revision, major revision, or rejection criteria.

Main Results:

• Gemini 2.5 Pro achieved 96.4% overall global suitability, with 27/28 items deemed suitable.
• ChatGPT-5 Thinking achieved 89.3% overall global suitability, with 25/28 items deemed suitable.
• Gemini 2.5 Pro scored 97.5% on the checklist, while ChatGPT-5 Thinking scored 89.0%.

Conclusions:

• LLMs can generate CoPJ items aligned with ethical declarations, with Gemini 2.5 Pro showing higher suitability.
• Further research is needed to evaluate LLM-generated CoPJ items in real learning environments and across diverse contexts.