Classification of Patients' Judgments of Their Physicians in Web-Based Written Reviews Using Natural Language...

Area of Science:

• Natural Language Processing
• Health Informatics
• Computational Linguistics

Background:

• Patients increasingly use web reviews for physician selection, but analyzing this unstructured text is challenging.
• Traditional methods like hand-coding and dictionaries have limitations in accuracy and scale.
• Advanced natural language processing (NLP) offers a solution for analyzing large volumes of physician reviews.

Purpose of the Study:

• To develop and validate NLP algorithms for classifying patient judgments in web-based physician reviews.
• To specifically identify the presence and sentiment (valence) of judgments regarding interpersonal manner and technical competence.
• To overcome limitations of previous methods in analyzing patient feedback at scale.

Main Methods:

• Trained and tested a transformer classification algorithm, Robustly Optimized BERT Pretraining Approach (RoBERTa), on 2000 hand-coded physician reviews.
• Developed two fine-tuned RoBERTa models to classify interpersonal manner and technical competence judgments.
• Validated model performance against 200 hand-coded reviews and the full dataset of 345,053 reviews.

Main Results:

• The interpersonal manner model achieved 90% accuracy (F1-score 0.89).
• The technical competence model achieved 90% accuracy (F1-score 0.90).
• Positive judgments correlated with higher star ratings, negative judgments with lower ratings, aligning with prior research.

Conclusions:

• RoBERTa models accurately classify patient judgments on physician interpersonal manner and technical competence at scale.
• The validated models demonstrate the potential of advanced NLP for analyzing unstructured review data.
• Future applications could extend to social media and electronic health records.