BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence
• Database Management

Background:

• Biomedical researchers require advanced data analysis tools for large structured databases.
• Existing text-to-SQL systems lack the domain reasoning needed for complex scientific queries.
• Bridging the gap between natural language questions and executable SQL is crucial for scientific discovery.

Purpose of the Study:

• Introduce BiomedSQL, the first benchmark for evaluating scientific reasoning in text-to-SQL generation.
• Assess the performance of large language models (LLMs) on complex biomedical queries.
• Provide a foundation for developing AI systems that support scientific discovery over structured biomedical data.

Main Methods:

• Developed BiomedSQL, a benchmark with 68,000 question/SQL query/answer triples.
• Grounded the benchmark in a harmonized BigQuery knowledge base integrating gene-disease, omics, and drug data.
• Evaluated various LLMs using different prompting strategies and interaction paradigms.

Main Results:

• Current LLMs show a significant performance gap in executing scientific queries.
• GPT-3-mini achieved 59.0% accuracy, while the custom BMSQL agent reached 62.6%.
• Both models performed substantially below the expert baseline accuracy of 90.0%.

Conclusions:

• BiomedSQL effectively highlights the limitations of current text-to-SQL models in scientific reasoning.
• There is a substantial need for AI systems with enhanced domain-specific reasoning capabilities.
• The BiomedSQL dataset and code will foster advancements in AI for biomedical research.