Predicting Semantic Similarity Between Clinical Sentence Pairs Using Transformer Models: Evaluation and...

Area of Science:

• Natural Language Processing (NLP)
• Clinical Informatics
• Machine Learning

Background:

• Semantic Textual Similarity (STS) is challenging in clinical text due to specialized language and abbreviations.
• Developing NLP systems for clinical STS requires robust models that understand nuanced medical terminology.

Purpose of the Study:

• To create an NLP system for predicting similarity scores between clinical sentence pairs.
• To analyze intermediate token vectors in transformer models to understand how semantic similarity is represented.

Main Methods:

• Averaged similarity scores from independently fine-tuned transformer models for clinical sentence pairs.
• Investigated model loss, decodability, and representational similarity of token vectors.

Main Results:

• Achieved a 0.87 correlation with ground-truth scores, ranking 6th out of 33 teams.
• Identified failure cases in modeling similarity for prescription details and overprediction with token overlap.
• Revealed divergent representational strategies between BERT and XLNet models.

Conclusions:

• Developed a competitive deep learning baseline for clinical STS without hand-crafted rules.
• Provided a detailed analysis of model outputs and learned biases in transformer models.
• Highlighted new research directions in model distillation and sentence embedding for clinical NLP.