Utilizing image and caption information for biomedical document classification

Area of Science:

• Biomedical Informatics
• Computational Biology
• Scientific Literature Analysis

Background:

• Biomedical research findings are disseminated through publications, necessitating efficient literature curation for databases.
• Manual curation is labor-intensive, making automated identification of relevant articles crucial for accelerating research.
• Current methods primarily rely on textual content from titles and abstracts, often overlooking valuable image and caption information.

Purpose of the Study:

• To develop an automated document classification scheme incorporating image and caption data alongside traditional text.
• To introduce a novel image representation, "Figure-word", for effective utilization of visual information.
• To enhance the accuracy and efficiency of identifying relevant biomedical publications for database curation.

Main Methods:

• A new document classification scheme was developed, integrating information from titles-and-abstracts, captions, and images.
• A novel image representation, "Figure-word", was introduced, derived from subfigure class labels.
• Word embeddings were used for textual data (captions and titles-and-abstracts).
• Two information integration methods were explored: feature vector combination and meta-classification.

Main Results:

• The proposed "Figure-word" representation effectively captures image information for document classification.
• Captions and titles-and-abstracts provide complementary information, enhancing classification performance.
• Combining "Figure-words", captions, and titles-and-abstracts led to significantly improved overall document classification accuracy.
• The study demonstrates the value of multimodal information (images, captions, text) in biomedical literature analysis.

Conclusions:

• The integration of image and caption data, represented by "Figure-words" and word embeddings, substantially improves biomedical document classification.
• This multimodal approach offers a more comprehensive and accurate method for identifying relevant literature compared to text-only methods.
• The developed scheme expedites the biocuration process, thereby supporting and accelerating biomedical research.