Evaluating Encoder and Decoder Models for Extended Clinical Concept Recognition in Japanese Clinical Texts: A...

The digitization of medical documents has resulted in vast amounts of information being stored electronically. Extracting medical knowledge for secondary purposes, such as diagnostic support, continues to pose a substantial challenge. While conventional named entity recognition (NER) has focused on short terms (e.g., genes, diseases, chemicals), the extraction and assessment of longer, complex expressions remain underexplored. Clinically vital concepts, such as diseases, pathologies, symptoms, and findings, often manifest as long phrases, whose accurate extraction is crucial for advanced applications like constructing causal knowledge from case reports. Consequently, a comprehensive framework addressing both short-term and clinically meaningful long phrase units-termed extended Clinical Concept Recognition (E-CCR)-is essential.

This study, the first comprehensive investigation into E-CCR model selection, aimed to identify optimal strategies by comparing encoder versus decoder models and general-purpose versus domain-specific pretraining. We also analyzed variations in model effectiveness with respect to target length and proposed a novel E-CCR evaluation metric.

We evaluated the extraction performance of 17 encoder and decoder models using the J-CaseMap database, which comprises approximately 20,000 case reports from Japan annotated with clinical concepts. Performance was primarily assessed using our novel "weighted soft matching score," which distinctively penalizes the fragmentation of long extraction targets and weights scores by target length to account for the increased difficulty of extracting longer expressions.

On the J-CaseMap dataset, JMedDeBERTa(s)-an encoder model pretrained on domain-specific medical text-achieved the highest mean performance (F1 = 0.758 (SD 0.002)), with similarly strong results from JMedDeBERTa(c), suggesting comparable performance among the top encoder models. As the fragmentation penalty increased, performance generally declined; however, no consistently severe degradation was observed. On the MRNER-disease dataset, the general-domain DeBERTaV2-base yielded the highest mean F1, and the differences among the medical-domain JMedDeBERTa(s)/(c) variants were small, suggesting that the benefit of domain-specific pretraining was limited. Overall, under our experimental setting (LoRA fine-tuning for decoders and full fine-tuning for encoders), encoder models outperformed decoder models, and token classification showed higher extraction performance than our instruction-tuning setup.

Under our experimental setting, for the E-CCR task, encoder-based token classification achieved the highest mean performance on our internal dataset. Differences among the top encoder models were small and should be interpreted as comparable within the uncertainty implied by our annotation review, whereas decoder-based approaches did not surpass encoder-based ones in this setup, suggesting that encoder models can deliver high accuracy with fewer parameters and may offer practical advantages in resource-constrained environments. In addition, token classification showed higher performance than our instruction-tuning setup for extracting long expressions, whereas instruction tuning was better suited to short terms. Using the weighted soft-matching score, we further found that performance did not substantially deteriorate as the fragmentation penalty increased, indicating that extracted spans were rarely fragmented. Similar trends were observed in external validation datasets, suggesting that the trends observed under our setup may generalize to information-extraction tasks on Japanese medical text. Nevertheless, further investigation is needed to determine whether these findings hold across other languages and medical document types.