Exploration of Genetic Entity Extraction From Spanish Literature Using Generative LLMs

The organization of information about genes, genetic variants, and associated diseases from scientific literature texts can facilitate progress in precision medicine. However, the vast scale of this literature demands development of automated strategies for identifying and extracting this information. Generative large language models (LLMs) represent a promising avenue for such automation. We systematically evaluate the performance of LLMs on the extraction of information relating to impacts of genetic variation on disease from the biomedical literature, considering the challenge of genetic and disease named entity recognition in Spanish-language scientific abstracts, and experimenting with a range of instruction strategies over a dataset known as GenoVarDis. We evaluate cross-linguistic prompting, and zero- and few-shot strategies, along with the optional provision of an annotation guideline and variations in the requested output format. A key finding is that the natural language of the prompt had only a limited impact on the model's performance on NER, demonstrating the feasibility of cross-linguistic information extraction. Overall, optimal results were obtained with few-shot prompting. However, we identify that generative LLMs failed to adhere to the instructions provided, leading to the over-generation and fabrication (hallucinations) of entities not appearing in the texts. We find that adding examples to the prompts and providing an overview of the expected output structure reduces hallucinated entities. Lastly, we explore the limitations of the prompting strategies and demonstrate the value of grounding generated outputs in the original texts. Overall, LLMs do not reach the accuracy of task-specific models, but we gain insight into effective strategies for their use.