Efficient information extraction using LLMs and knowledge distillation: A study on HPV health communication.

Area of Science:

• Computational Linguistics
• Public Health Informatics
• Health Communication

Background:

• State Department of Health (DOH) websites are key sources for human papillomavirus (HPV) health information, influencing public awareness and vaccination decisions.
• Systematic evaluation of the quality, completeness, and motivational impact of HPV vaccination information on these websites is crucial but challenging.

Purpose of the Study:

• To develop a computationally efficient framework for evaluating the quality and impact of HPV vaccination information on State DOH websites.
• To create a consolidated dataset from 48 DOH websites and an annotated dataset for model training and evaluation.

Main Methods:

• Development of a Knowledge Distillation framework using Large Language Models (LLMs) and RoBERTa Large.
• Creation of an annotated dataset (n=400) from 48 State DOH websites focusing on HPV and HPV vaccination.
• Training and evaluation of efficient student models, including fine-tuned RoBERTa Large, for content assessment.

Main Results:

• The fine-tuned RoBERTa Large model achieved a high F1 score of 0.74 on the test set, outperforming other student models.
• The best-performing model demonstrated performance close to the teacher model (F1=0.77).
• The model was successfully deployed for computationally feasible evaluation of State DOH website content.

Conclusions:

• A fine-tuned RoBERTa Large model provides an effective and efficient method for evaluating the quality of HPV vaccination information on State DOH websites.
• This framework supports systematic assessment, potentially improving public health communication strategies regarding HPV vaccination.
• The study highlights the utility of LLMs and distillation for analyzing health information repositories and discusses broader implications.