Detecting Role Impersonation in AI-Generated Clinical Oncology Text Using Knowledge Graphs

Area of Science:

• Artificial Intelligence in Medicine
• Oncology Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly used in oncology for clinical documentation.
• LLMs' fluency can mask a risk of impersonating oncologists without true oncologic reasoning.
• Existing methods lack domain-specific checks for AI-generated oncology content.

Purpose of the Study:

• To present a framework for detecting domain-level role impersonation in AI-generated oncology communication.
• To quantify semantic fidelity and procedural coherence in AI-generated clinical notes.
• To safeguard the trustworthy integration of generative AI in oncology practice.

Main Methods:

• Developed a role impersonation index (RII) using semantic fidelity and procedural coherence (PC).
• Aligned generated content with expert knowledge from ontologies (SNOMED CT, UMLS, NCIt, OncoTree).
• Evaluated on 4,800 clinical oncology notes (human vs. AI authorship).

Main Results:

• The framework detects inconsistencies in tumor terminology, staging, biomarker logic, and therapeutic sequencing.
• Achieved a 94% F1-score and 0.96 AUROC in distinguishing human from AI authorship.
• Successfully identified impersonation without relying on superficial linguistic cues.

Conclusions:

• The proposed framework provides a principled safeguard against AI role impersonation in oncology.
• Integrating knowledge-graph alignment with procedural logic scoring is novel for detecting impersonation.
• This approach supports the reliable use of generative AI in cancer care documentation.