Large language models with retrieval-augmented generation enhance expert modelling of Bayesian network for clinical...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems
• Bioinformatics and Computational Biology

Background:

• Bayesian networks (BNs) offer transparent and interpretable models crucial for clinical decision support.
• Traditional BN modeling is labor-intensive, requiring significant manual effort and expertise.
• Enhancing BN modeling efficiency and accuracy is vital for broader clinical adoption.

Purpose of the Study:

• To investigate the integration of large language models (LLMs) with retrieval-augmented generation (RAG) to improve Bayesian network (BN) modeling.
• To assess the impact of LLM-RAG on the efficiency, accuracy, and cognitive workload associated with BN model creation.
• To evaluate the clinical relevance and usability of an AI-assisted BN modeling service.

Main Methods:

• Development of a web-based BN modeling service incorporating an LLM-RAG pipeline.
• Utilized a fine-tuned GTE-Large embedding model for knowledge retrieval, optimized with recursive chunking and query expansion.
• Employed GPT-4 and Mixtral 8x7B for data interpretation and suggestion generation, respectively.
• Conducted a user study with clinicians to assess usability, retrieval accuracy, and cognitive workload via NASA-TLX.

Main Results:

• The LLM-RAG pipeline demonstrated improved retrieval accuracy (up to 0.9) and answer relevance.
• Optimized retrieval techniques enhanced accuracy for semantic chunking (0.75 to 0.85) with high response faithfulness (≥0.9).
• Clinicians successfully created comprehensive BN models within an hour, with the tool reducing cognitive workload (2/7 NASA-TLX).
• While intuitive, the system occasionally struggled with adherence to predefined causal structures, and minor technical issues were noted.

Conclusions:

• LLM-RAG integration significantly enhances the efficiency and accuracy of Bayesian network modeling.
• Future work should focus on automated preprocessing, UI refinement, and expanding RAG with validation and external data sources.
• Generative AI presents a promising avenue for advancing expert-driven knowledge modeling in clinical settings.