Large Language Model-Based Text-to-Cypher Generation for Question-Answering on Biomedical Knowledge Graphs

Knowledge graphs are widely adopted due to their flexible structure and ability to represent large-scale, heterogeneous data. However, as these graphs grow, the queries used to retrieve information become more complex, making efficient information retrieval and question answering more challenging.

This work investigates (1) the impact of few-shot versus zero-shot prompting on Cypher query generation, and (2) performance differences between small (∼7B) and mid-size (∼32B) large language models (LLMs) on both of these approaches.

We propose a system that translates natural language questions into Cypher queries for execution on a Neo4j knowledge graph and evaluates answer quality on the PrimeKG and BioHopR datasets. To address data protection and cost constraints, we focus on locally deployable LLMs that require no additional training, using zero-shot and few-shot prompting strategies.

32B models generate syntactically valid Cypher queries using only the knowledge graph schema information. Few-shot prompting with four examples improves F1 score by 35.7 percentage points (∼62% relative improvement) compared to zero-shot prompting. While 7B models struggle to generate syntactically valid queries in the zero-shot setting, few-shot prompting substantially improves their performance, enabling high query syntax validity and F1 scores above 80%.

Few-shot prompting heavily improves query generation quality and enables effective use of smaller models, supporting the practical deployment of locally hosted LLMs for knowledge graph question answering.