Text summarization for pharmaceutical sciences using hierarchical clustering with a weighted evaluation methodology.

Area of Science:

• Pharmaceutical Regulatory Science
• Computational Linguistics
• Machine Learning Applications

Background:

• Pharmaceutical industry relies heavily on extensive regulatory documents.
• Large document size poses challenges for efficient review and decision-making.
• Need for automated summarization tools to improve accessibility.

Purpose of the Study:

• To introduce MedicoVerse, a novel machine learning solution for summarizing large regulatory documents.
• To evaluate MedicoVerse's performance against established summarization models.
• To provide project teams with informative and concise document summaries.

Main Methods:

• Utilized SapBERT for word embeddings on regulatory documents.
• Applied hierarchical agglomerative clustering to organize embeddings.
• Employed bart-large-cnn-samsum for cluster summarization.
• Developed a custom data structure for cluster organization.

Main Results:

• MedicoVerse demonstrated superior performance compared to T5, Google Pegasus, Facebook BART, Mixtral, GPT 3.5, and Llama-2-70b.
• Evaluation metrics included ROUGE score, BERTScore, business entities, and Flesch Reading Ease.
• The system successfully generated comprehensive summaries of large regulatory documents.

Conclusions:

• MedicoVerse provides an effective and efficient method for summarizing complex pharmaceutical regulatory texts.
• The novel multi-stage approach enhances the utility of regulatory document analysis.
• This tool can significantly aid project teams in navigating the regulatory landscape.