Iterative Prompt Refinement for Mining Gene Relationships from ChatGPT

ChatGPT has demonstrated its potential as a surrogate knowledge graph. Trained on extensive data sources, including open-access publications, peer-reviewed research articles and biomedical websites, ChatGPT extracted information on gene relationships and biological pathways. However, a major challenge is model hallucination, i.e., high false positive rates. To assess and address this challenge, we systematically evaluated ChatGPT's capacity for predicting gene relationships using GPT-3.5-turbo and GPT-4. Benchmarking against the KEGG Pathway Database as the ground truth, we experimented with diverse prompting strategies, targeting gene relationships of activation, inhibition, and phosphorylation. We introduced an innovative iterative prompt refinement technique. By assessing prompt efficacy using metrics like F-1 score, precision, and recall, GPT-4 was re-engaged to suggest improved prompts. A refined prompt, which combines a specialized role with explanatory text, significantly enhances the performance. Going beyond pairwise gene relationships, we also deciphered complex gene interplays, such as gene interaction chains and pathways pertinent to diseases like non-small cell lung cancer. Direct prompts showed limited success, but "least-to-most" prompting exhibited significant potentials for such network constructions. The methods in this study may be used for some other bioinformatics prediction problems.