Geriatric syndromes extraction from discharge summaries: a new dataset, annotation scheme and initial findings

Geriatric syndromes (GS), such as falls, dementia, delirium and malnutrition, are complex clinical conditions affecting older adults which involve multiple organ systems and have major impact on quality of life and care. GS cut across disease categories, and are poorly represented in structured electronic health records. Natural language processing (NLP) offers an opportunity to extract valuable GS-related information from unstructured clinical text, such as hospital discharge summaries. However, the lack of high-quality annotated datasets limits the effectiveness of NLP models in this domain. This study introduces a manually annotated corpus designed for GS detection, enabling more accurate identification and classification of GS.

We developed a comprehensive and detailed annotation scheme to label 12 common GS from hospital discharge summaries, incorporating key attributes such as diagnosis type, negation and event occurrence. The corpus consists of 2,040 manually annotated discharge summaries from National Health Service (NHS) Lothian hospitals in Scotland. To assess the effectiveness of NLP in extracting GS, we experimented with multiple pretrained transformer-based models, including base BERT (general-domain), BioBERT (biomedical-domain), BioClinicalBERT (clinical-domain) and BERT-cased (the cased English BERT checkpoint). The models were fine-tuned and tested for two types of tasks: named entity recognition (NER) and document-level labelling. We also considered an extra task of detecting contextual information with each GS mention (e.g., history, suspected, in-hospital). When context information is considered, two new tasks are called NER-C and DL-C, for NER and document-level labelling with context respectively.

Our evaluation showed that, for the document-level labelling task, BERT-cased achieved the highest F1-score (0.897) and BioClinicalBERT performed best when negation was considered (F1-score: 0.888). For the NER task, BioClinicalBERT and BERT-cased achieved an F1-score of 0.883. Frailty (F1 $=$ 1.0), Falls (F1 $=$ 0.973) and Delirium (F1 $=$ 0.946) are the GS entities with the best performing results. For NER-C, BERT-cased achieved the best F1 of 0.692 and BioBERT performed the worst (F1 $=$ 0.658). In NER-C, the best results were achieved for context-aware falls and frailty labels, particularly when the syndrome was implied rather than explicitly stated. Document-level aggregation helped reduce inconsistencies, but the NER experiments used a flattened CoNLL-compatible representation of the original annotations, in which discontinuous mentions were converted into shortest covering spans and overlapping mentions were merged. Therefore, the reported NER results should be interpreted as baseline performance on a simplified representation of these structures, while low-frequency GS categories and sparse contextual labels also negatively affected model accuracy.

This study demonstrates the effectiveness of NLP for extracting geriatric syndromes from unstructured clinical text and introduces a manually annotated corpus with detailed guidelines to support this task. The results also show that model performance is strongly shaped by dataset characteristics. More frequent and lexically clearer syndromes, such as frailty, falls and delirium, achieved the strongest results, whereas rarer categories and low-frequency attribute combinations, such as suspected, referral and some negated or context-specific labels, were harder to learn and yielded lower and less stable scores. Likewise, fine-grained annotation was more challenging than coarse-grained annotation because it increases label sparsity and requires the model to distinguish subtle contextual differences, such as current vs. historical mentions, implicit mentions and in-hospital onset. Entity-level extraction was further affected by discontinuous and overlapping mentions, which are common in clinical narratives and make boundary detection harder, whereas document-level aggregation reduced the impact of these local errors and therefore produced higher scores. These findings underline that data distribution, annotation complexity and mention structure directly influence model performance, and should be central considerations in future work on geriatric syndrome extraction.