A Fully Expert Human-Based Retrieval Augmented Generation (FEH-RAG) Framework: A Proof of Concept Study in Labelling...

Accurate application of reference standards (RSs) is essential for correct decision-making in areas governed by such standards. Yet in real-world practice, even fully trained users often apply RSs inconsistently, due to cognitive overload, stress, or other contextual factors, generating misleading evidence. This problem is exemplified by the fact that up to 80% of board-certified hematologists mislabel patients with Sjögren syndrome (SS), a connective tissue disorder (CTD) associated with the greatest risk of lymphoproliferative disorders compared to other CTDs. Embedding RS-based consultations with full objectivity into the decision-making layers of digitalized and non-digitalized settings is critical for improving both decision-making and the quality of resulting evidence. The aim of this proof-of-concept (POC) study is to apply a newly developed framework, the Fully Expert Human-based Retrieval Augmented Generation (FEH-RAG) framework, to provide such a foundation for augmenting SS case labeling in routine daily clinical practice.

In this POC study, using the nine steps of the FEH-RAG framework, seven expert end-users systematically selected the most widely used SS classification criteria (SSCC) and extracted their elements, including items, definitions, item weights, and inter-item relationships. Extracted items were profiled based on their usage in routine clinical practice. A pathway layout and decision tables were developed accordingly. Following pathway generation, the residual misalignment of the outputs with the SSCC was assessed in a cohort of patients at risk of SS. The experts predefined that the residual misalignment rate of the FEH-RAG outputs with the SSCC must be ≤2% (95% confidence, using the rule of three).

The FEH-RAG framework objectively generated RS-based, transparent, and traceable outputs, including decision tables, an SS classification pathway, and a list of misinterpretations of the SSCC. These misinterpretations involved definitions of dry eye and dry mouth, application of secondary SS criteria, handling SS criteria-specific exclusion rules, and interpretation of serological and objective test results. This POC study achieved its expert-defined maximum misalignment threshold of ≤2% with 95% confidence (0 misalignment in 150 consecutive patients at risk of SS).

This POC study established the needed foundation for improving SS case labeling in daily clinical practice across both digital and non-digital settings. As shown here, publishing FEH-RAG outputs while highlighting potential RS misinterpretations offers a transparent and traceable basis for augmenting decision-making in domains governed by RSs.