CACER: Clinical concept Annotations for Cancer Events and Relations

Area of Science:

• Natural Language Processing in Oncology
• Biomedical Informatics
• Clinical Data Mining

Background:

• Clinical notes contain unstructured patient histories, including relationships between medical problems and prescription drugs.
• Extracting structured semantic representations is crucial for understanding drug-problem associations and symptom burden in cancer patients.

Purpose of the Study:

• To investigate the relationship between cancer drugs and their associated symptom burden by extracting structured information from oncology notes.
• To develop and evaluate transformer-based information extraction models using a novel annotated corpus.

Main Methods:

• Creation of the Clinical concept Annotations for Cancer Events and Relations (CACER) corpus with over 48,000 annotations for medical problems and drug events, and 10,000 relations.
• Development and evaluation of transformer models including Bidirectional Encoder Representations from Transformers (BERT), Fine-tuned Language Net Text-To-Text Transfer Transformer (Flan-T5), Large Language Model Meta AI (Llama3), and Generative Pre-trained Transformers-4 (GPT-4).
• Utilizing fine-tuning and in-context learning (ICL) approaches for model evaluation.

Main Results:

• Fine-tuned BERT and Llama3 models achieved the highest performance in event extraction (88.2-88.0 F1), comparable to inter-annotator agreement (88.4 F1).
• Fine-tuned BERT, Flan-T5, and Llama3 models showed the highest performance in relation extraction (61.8-65.3 F1).
• GPT-4 with ICL performed the worst across both event and relation extraction tasks.

Conclusions:

• Fine-tuned models significantly outperformed GPT-4 with ICL, emphasizing the value of annotated data and model optimization.
• BERT models demonstrated performance similar to Llama3, indicating no significant advantage for larger language models in this specific task.
• The CACER corpus and evaluated models provide a foundation for structured information extraction from oncology clinical narratives.