Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge:...

Area of Science:

• Artificial Intelligence
• Health Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) demonstrate advanced capabilities in general knowledge but have limited evaluation within the specialized Fast Healthcare Interoperability Resources (FHIR) standard.
• The complexity of FHIR poses challenges for LLMs trained on broad datasets, potentially limiting their understanding of domain-specific healthcare data.
• Enhancing health data interoperability is crucial for leveraging clinical data and improving electronic health record interactions.

Purpose of the Study:

• Introduce the FHIR Workbench, a novel dataset suite designed to rigorously assess LLM comprehension and application of the FHIR standard.
• Evaluate the performance of both open-source and commercial LLMs on a variety of FHIR-related tasks.
• Provide a benchmark for understanding LLM capabilities in healthcare data interoperability.

Main Methods:

• Developed four distinct evaluation datasets to test FHIR knowledge, including multiple-choice questions on FHIR concepts and the FHIR Representational State Transfer (REST) API.
• Assessed LLM performance in a zero-shot setting on tasks such as identifying FHIR resource types and generating FHIR resources from unstructured clinical notes.
• Included human evaluations with six participants of varying FHIR expertise to contextualize LLM performance metrics.

Main Results:

• Commercial models like GPT-4o achieved high F1-scores (e.g., 0.9990 on FHIR-ResourceID), while open-source models like DeepSeek-v3 also showed strong results (e.g., 0.9400 on FHIR-QA).
• All evaluated LLMs struggled with the Note2FHIR task, indicating significant challenges in converting unstructured clinical text into FHIR-compliant resources, with scores ranging from 0.0382 to 0.3633.
• Human participants achieved accuracy scores between 0.50 and 1.0 on the initial three FHIR tasks.

Conclusions:

• Both open-source and commercial LLMs exhibit competitive performance on FHIR-related tasks, with commercial models currently leading in complex applications.
• The FHIR Workbench serves as a critical tool for evaluating LLM capabilities and driving advancements in health data interoperability.
• Continued development is needed to address the challenges LLMs face in accurately processing and converting unstructured clinical data into standardized FHIR resources.