Microbial named entity recognition and normalisation for AI-assisted literature review and meta-analysis

Manual curation of biomedical literature is slow and error-prone and while large language models trained on general texts have shown to be useful for text summarisation, these methods lack the domain-specific expertise required to perform this task accurately. Here we describe the creation of the first microbiome-specific text corpus, use this to train deep learning algorithms for named-entity recognition (NER) and entity linking (EL), and demonstrate their use to meta-analyse microbiome literature.

The training and validation set (n = 1410) contained a total of 90 150 annotations (both long form and abbreviations). Using the gold-standard test set (n = 288), with an inter-annotator agreement rate of 99.52% for NER and 88.31% for EL, the trained models were evaluated and our fine-tuned BioBERT model achieved an F1-score of 96% for NER surpassing a rule- and dictionary-based annotation pipeline (94%). For EL the accuracy obtained by the deep learning models greatly surpassed that of the pipeline (91% vs 69%). Evaluated across the entire available literature (n = 6927) across 14 domains, our models annotate an entire full-text document in only 7 seconds.

All codes are available for automatic annotation and model training, with instructions on how to deploy the model on new text, from GitHub at https://github.com/omicsNLP/microbELP and Zenodo at https://dx.doi.org/10.5281/zenodo.20613467. The redistributable, annotated training set and unannotated test set are made available from Zenodo at https://dx.doi.org/10.5281/zenodo.17305410 with the redistributable, human-labelled test set hosted as benchmark on Codabench at https://www.codabench.org/competitions/10913/ (for NER only) and at https://www.codabench.org/competitions/11581/ (for NER+EL) for evaluation. The annotated documents for all available literature are hosted separately on Zenodo at https://dx.doi.org/10.5281/zenodo.17288826.