Semantic Academic Profiler (SAP): a framework for researcher assessment based on semantic topic modeling.

Area of Science:

• Computer Science
• Information Science
• Bibliometrics

Background:

• Current methods for identifying research expertise and co-authorship networks rely on topic modeling but lack real-world quality evaluations.
• Existing approaches do not adequately assess the quality of generated topics or provide robust methods for profile validation.

Purpose of the Study:

• To introduce the Semantic Academic Profiler (SAP) framework for summarizing research articles, automatically building researcher profiles, and enabling online quality assessments.
• To evaluate the effectiveness of SAP in summarizing entities and supporting online topic quality assessments.
• To compare different topic modeling strategies and data representation techniques for profile generation.

Main Methods:

• Developed the Semantic Academic Profiler (SAP) framework, extending Cluwords topic modeling with n-grams and a visual interface.
• Conducted an offline evaluation using Normalized Pointwise Mutual Information (NPMI) to compare TFIDF and Cluwords with and without bi-grams on Brazilian curricula vitae.
• Performed an online evaluation via an A/B test where researchers assessed their generated profiles and a usability test (SUS) for the SAP interface.

Main Results:

• Cluwords with bi-grams emerged as the superior strategy for topic modeling and profile generation.
• The SAP interface was found to be highly useful for researchers.
• Significant discrepancies were observed between online and offline evaluation results, highlighting the need for both assessment types.

Conclusions:

• The Semantic Academic Profiler (SAP) framework, particularly with Cluwords and bi-grams, offers a robust solution for generating and evaluating research profiles.
• Integrating both online and offline evaluation methods provides a comprehensive assessment of topic quality and profile relevance.
• This study underscores the importance of real-world validation in topic modeling research and opens avenues for future investigations.