ODD: A Benchmark Dataset for the Natural Language Processing Based Opioid Related Aberrant Behavior Detection.

Area of Science:

• Biomedical Natural Language Processing
• Clinical Informatics
• Opioid Use Disorder Research

Background:

• Opioid-related aberrant behaviors (ORABs) are significant risk factors for opioid overdose.
• Accurate identification of ORABs from clinical notes is crucial for patient safety.
• Existing methods for ORAB detection lack comprehensive datasets.

Purpose of the Study:

• Introduce the Opioid-related Aberrant Behavior Detection (ODD) dataset, a novel biomedical NLP resource.
• Facilitate the development and evaluation of NLP models for ORAB identification.
• Classify ORABs into nine distinct categories within electronic health records.

Main Methods:

• Developed an expert-annotated dataset (ODD) comprising patients' EHR notes.
• Utilized two state-of-the-art NLP approaches: fine-tuning and prompt-tuning.
• Evaluated model performance across nine ORAB categories using macro average area under the precision-recall curve.

Main Results:

• Prompt-tuning NLP models generally outperformed fine-tuning models in ORAB detection.
• Performance gains were most pronounced for identifying less common ORAB categories.
• The best model achieved a macro average area under the precision-recall curve of 88.17%.

Conclusions:

• The ODD dataset provides a valuable resource for advancing ORAB detection research.
• Prompt-tuning demonstrates superior performance for identifying diverse ORABs, including rare instances.
• Further improvements are needed for enhanced detection of uncommon ORAB categories.