An LLM-Powered Clinical Calculator Chatbot Backed by Verifiable Clinical Calculators and their Metadata.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Clinical calculators are essential tools in healthcare.
• Large language models (LLMs) offer new ways to interact with these tools using natural language.

Purpose of the Study:

• To develop and evaluate a purpose-built chatbot for clinical calculators.
• To assess the chatbot's accuracy in metadata interrogation and clinical calculations compared to an unassisted LLM.

Main Methods:

• The chatbot integrates software implementations of clinical calculators with LLM tools.
• Retrieval augmented generation (RAG) is used for accessing calculator metadata.
• Accuracy was compared against an unassisted LLM across four natural language conversation workloads.

Main Results:

• The chatbot achieved 100% accuracy for queries about calculator metadata.
• Clinical calculation accuracy improved significantly: 86.4% (complete sentences) and 79.2% (shorthand) vs. 61.8% and 62.0% for the unassisted LLM.
• Calculation errors were eliminated for complete sentences (0% vs. 16.8%) and reduced for shorthand (2.4% vs. 18%).

Conclusions:

• The developed chatbot demonstrates enhanced accuracy for clinical calculations and metadata retrieval.
• While not yet ready for clinical deployment, the chatbot shows significant progress in reducing calculation errors.
• This approach highlights the potential of LLMs and RAG for improving the reliability of clinical decision support tools.