A Fully Expert Human-Based Retrieval Augmented Generation (FEH-RAG) Framework: A Proof of Concept Study in Labelling...

Area of Science:

• Medical Diagnostics
• Clinical Decision Support Systems
• Rheumatology

Background:

• Inconsistent application of reference standards (RSs) leads to diagnostic errors, exemplified by up to 80% of hematologists mislabeling Sjögren syndrome (SS).
• SS is a connective tissue disorder (CTD) with a high risk of lymphoproliferative disorders, necessitating accurate diagnosis.
• Objective, RS-based decision support is crucial for improving diagnostic accuracy and evidence quality in clinical practice.

Purpose of the Study:

• To introduce and evaluate the Fully Expert Human-based Retrieval Augmented Generation (FEH-RAG) framework as a proof-of-concept (POC) for augmenting SS case labeling.
• To provide an objective and traceable foundation for decision-making in both digital and non-digital clinical settings.

Main Methods:

• The FEH-RAG framework was applied using nine steps, involving expert end-users to select SS classification criteria (SSCC) and extract their components.
• Decision pathways and tables were developed based on extracted SSCC elements, profiling item usage in routine clinical practice.
• Residual misalignment between FEH-RAG outputs and SSCC was assessed, with an expert-defined threshold of ≤2% misalignment.

Main Results:

• The FEH-RAG framework generated objective, transparent, and traceable outputs, including decision tables and an SS classification pathway.
• Identified common misinterpretations of SSCC related to dry eye/mouth definitions, secondary SS criteria, exclusion rules, and test result interpretation.
• Achieved 0% misalignment in 150 consecutive patients at risk for SS, meeting the expert-defined threshold of ≤2% with 95% confidence.

Conclusions:

• The POC study established a foundation for enhancing SS case labeling accuracy in routine clinical practice.
• Publishing FEH-RAG outputs, including common misinterpretations, provides a transparent basis for decision support in RS-governed domains.
• The FEH-RAG framework demonstrates potential for improving diagnostic accuracy and reducing errors in complex medical conditions like SS.