Analysis of AI-Generated Radiography Responses Using a Closed-System LLM

To evaluate the accuracy and educational utility of Microsoft Copilot's (GPT-4, July 2025, closed-system version) responses to radiography questions through expert assessment, with a focus on strengths, limitations, and implications for radiologic science education.

This qualitative descriptive study evaluated Copilot's responses to 15 open-ended radiography questions derived from the American Registry of Radiologic Technologists Radiography Examination Content Specifications. Seven subject matter experts with extensive clinical and teaching experience independently reviewed the artificial intelligence (AI)-generated responses for accuracy and educational utility. Feedback was collected using Microsoft Forms and analyzed inductively following a 6-phase thematic analysis framework.

Thematic analysis revealed 6 overarching themes: accuracy and completeness of content, scope of practice and role clarification, outdated terminology and standards, formatting and presentation strengths, utility as a learning aid, and need for specificity and context. Experts praised the clarity, structure, and organization of responses and noted their potential as supplemental study aids. However, concerns were raised about incomplete or superficial content, attributions outside a radiologic technologist's scope of practice, outdated terminology and standards (including shielding and grid use), lack of specificity, and insufficient clinical context.

Findings suggested that although Copilot might provide structured and accessible support for radiography learners, its limitations could result in outdated or inaccurate practices if used without a critical lens. The closed-system design further constrained educational utility by preventing transparent sourcing. For radiography education, these results highlighted the importance of embedding critical AI literacy skills into curricula so that students learn to evaluate, verify, and contextualize AI-generated outputs.

Copilot demonstrated potential as a supplemental learning aid in radiography education, but outdated terminology, technical inaccuracies, and lack of sourcing constrained its reliability. Future research should compare multiple AI platforms, assess student learning outcomes, and explore strategies for embedding AI literacy and institutional safeguards to support safe, effective integration into health professions education.