LLM-in-the-Loop execution of clinical quality language

We present a method and open-source prototype to augment Clinical Quality Language (CQL) using a large language model (LLM) in the execution loop for querying unstructured text.

We modified a popular CQL engine to trigger an LLM-in-the-Loop pipeline (LitL) whenever CQL references unstructured FHIR resources, without altering the existing structured resource handling. LitL generates a binary response to natural language queries from CQL using an LLM.

We present an open-source prototype that includes: an implementation of the LitL pattern, and a modified CQL execution engine that triggers LitL for unstructured data. The feasibility testing with two locally hosted LLMs achieved an accuracy of 72% and 93% respectively. An end-to-end prototype is provided to facilitate implementation with configurable models, prompts and hyperparameters.

The LitL pattern extends CQL to unstructured data, preserving compatibility with existing CQL and reducing hallucination. All components of the prototype are free and open source.