TopicForest: embedding-driven hierarchical clustering and labeling for biomedical literature

Area of Science:

• Biomedical Informatics
• Computational Biology
• Data Science

Background:

• The rapid growth of biomedical literature presents challenges for organizing and interpreting complex research topics.
• Existing topic modeling methods often produce flat clusters, failing to capture the hierarchical nature of scientific subjects.
• There is a need for advanced techniques to model the intricate hierarchies within biomedical research.

Purpose of the Study:

• To develop TopicForest, an embedding-driven framework for hierarchical topic modeling in biomedical literature.
• To create a forest of topic trees, enabling exploration from broad areas to narrow specialties.
• To improve the organization and interpretation of complex biomedical research topics.

Main Methods:

• Utilized contrastively trained large language models (LLMs) to embed biomedical abstracts.
• Applied manifold learning for dimensionality reduction and visualization.
• Implemented hierarchical clustering using binary partitioning and dendrogram cutting.
• Employed recursive LLM-based summarization for topic labeling at multiple granularities.

Main Results:

• TopicForest achieved competitive or superior Adjusted Mutual Information (AMI) scores compared to flat clustering methods like BERTopic.
• The framework outperformed the deep hierarchical topic model HyperMiner in clustering performance.
• LLM-based recursive labeling demonstrated higher label diversity and hierarchical affinity than existing methods.
• TopicForest exhibited stable clustering quality across different embedding models, indicating robustness.

Conclusions:

• TopicForest offers an effective and interpretable approach to hierarchical topic modeling for biomedical literature.
• The integration of LLMs, dimension reduction, and hierarchical clustering facilitates multi-scale exploration and visualization of research corpora.
• This framework enhances the ability to navigate and understand the complex landscape of biomedical research.