Evaluating topic model interpretability from a primary care physician perspective.

Area of Science:

• Natural Language Processing
• Machine Learning in Healthcare
• Clinical Informatics

Background:

• Probabilistic topic models offer unsupervised text analysis.
• These models identify word combinations (topics) for potential clinical summarization.
• Human interpretability of topics from clinical reports is not well understood.

Purpose of the Study:

• Assess topic coherence and representativeness in clinical reports from a primary care physician's perspective.
• Evaluate the utility of topic models for clinical informatics applications.

Main Methods:

• Three latent Dirichlet allocation models (50, 100, 150 topics) were applied to clinical reports.
• Primary care physicians and graduate students manually evaluated topic interpretability.
• Statistical tests (Wilcoxon, Mann-Whitney U) compared model performance and rater groups.

Main Results:

• The 100-topic model showed superior granularity, with participants accurately identifying irrelevant words.
• All models demonstrated comparable document representation capabilities.
• Primary care physicians significantly outperformed students in evaluating topic coherence and relevance.

Conclusions:

• This study establishes a baseline for topic model interpretability in clinical text.
• Primary care physicians perceive discovered topics as more coherent and representative than students.
• Further research is warranted for using topic models in automatic clinical summarization.