LLM-in-the-Loop execution of clinical quality language

Area of Science:

• Health Informatics
• Natural Language Processing
• Clinical Decision Support

Background:

• Clinical Quality Language (CQL) is crucial for standardized clinical decision support.
• Querying unstructured clinical text remains a significant challenge in healthcare data analysis.
• Existing CQL engines primarily handle structured data, limiting their application to narrative clinical notes.

Purpose of the Study:

• To develop a method and open-source prototype for augmenting CQL with Large Language Models (LLMs).
• To enable CQL to query unstructured clinical text by integrating LLMs into the execution loop.
• To preserve compatibility with existing CQL infrastructure while expanding its capabilities.

Main Methods:

• Modified a popular CQL engine to incorporate an LLM-in-the-Loop (LitL) pipeline.
• The LitL pipeline is triggered when CQL references unstructured FHIR resources.
• LLMs generate binary responses to natural language queries derived from CQL.

Main Results:

• An open-source prototype implementing the LitL pattern and a modified CQL engine was developed.
• Feasibility testing with two local LLMs yielded accuracies of 72% and 93%.
• The prototype supports configurable models, prompts, and hyperparameters for flexible implementation.

Conclusions:

• The LitL pattern successfully extends CQL capabilities to unstructured data.
• This approach maintains compatibility with existing CQL standards and reduces LLM hallucination.
• The free and open-source nature of the prototype facilitates wider adoption and further development.