Related Experiment Video
Updated: Jun 23, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large language model chatbot-based text-to-SQL application for database analyses in liver diseases and hepatology
Aryana T Far1, Steve Sun2, Gabrielle Jutras1
1Division of Gastroenterology and Hepatology, Department of Medicine at the University of California, San Francisco, San Francisco, CA, United States.
JAMIA Open
|June 22, 2026
Summary
Large language models (LLMs) can power Text2SQL chatbots for natural language queries on clinical data. While promising for data access, performance varies by query type.
Area of Science:
- Artificial Intelligence in Medicine
- Clinical Data Analysis
- Natural Language Processing
Background:
- Structured clinical data analysis is crucial for research.
- Natural language queries can enhance data accessibility for clinicians.
- Large language models (LLMs) offer potential for interpreting natural language queries.
Purpose of the Study:
- To develop and evaluate a proof-of-concept Text2SQL chatbot for hepatocellular carcinoma (HCC) research.
- To enable PHI-compliant natural language interaction with structured clinical data.
- To assess the performance of an LLM-powered chatbot in translating natural language to SQL queries.
Main Methods:
- Developed an interactive, PHI-compliant Text2SQL chatbot on a secure AI platform.
- Configured the chatbot to translate natural language queries into SQL, execute them, and return results.
- Clinicians posed 30 questions, with outputs compared against data scientist-written SQL, assessing prompt interpretation, SQL accuracy, and output accuracy.
Main Results:
- Prompt interpretation accuracy was 73.3%.
- SQL accuracy ranged from 53% to 63%.
- Output accuracy ranged from 53% to 63%, with common errors in group comparisons, ambiguous variables, and exploratory questions.
Conclusions:
- Text2SQL facilitates PHI-compliant interaction with clinical datasets without requiring coding expertise.
- LLM-powered Text2SQL presents a viable and promising method for improving clinician access to clinical data.
- Performance variability across query types indicates areas for future improvement.
