Assessing domain adaptation in adverse drug event extraction on real-world breast cancer records

Adverse Drug Events (ADE) are key information present in unstructured portions of Electronic Health Records. These pose a significant challenge in healthcare, ranging from mild discomfort to severe complications, and can impact patient safety and treatment outcomes.

We explore the influence of domain shift between a set of dummy clinical notes and a real-world hospital corpus of Japanese clinical notes of breast cancer treatment when extracting ADEs from free text. We annotated a subset of the hospital dataset and used it to fine-tune a Named Entity Recognition (NER) model, initially trained with the set of dummy documents. We used increasing amounts of the annotated data and evaluated the impact on the model's performance. Additionally, we examined the extracted information to identify combinations of drugs that are likely to cause ADEs.

We show that domain adaptation can significantly improve model performance in the new domain, as by feeding a small subset of 100 documents for the fine-tuning process we saw a 40% improvement in model performance. However, we also noticed diminishing returns when fine-tuning the model with a larger dataset. For instance, by feeding eight times more data, we only saw further 18% improvement in extraction performance.

While variations in writing style and vocabulary in clinical corpora can significantly impact the quality of NER results. We show that domain adaptation can be of great aid in mitigating these discrepancies and achieving better performance. Yet, while providing in-domain data to a model helps, there are diminishing returns when fine-tuning with large amounts of data.