Combining Deep Learning and Knowledge-driven Reasoning for Chest X-Ray Findings Detection

The application of deep learning algorithms in medical imaging analysis is a steadily growing research area. While deep learning methods are thriving in the medical domain, they seldom utilize the rich knowledge associated with connected radiology reports. The knowledge derived from these reports can be utilized to enhance the performance of deep learning models. In this work, we used a comprehensive chest X-ray findings vocabulary to automatically annotate an extensive collection of chest X-rays using associated radiology reports and a vocabulary-driven concept annotation algorithm. The annotated X-rays are used to train a deep neural network classifier for finding detection. Finally, we developed a knowledge-driven reasoning algorithm that leverages knowledge learned from X-ray reports to improve upon the deep learning module's performance on finding detection. Our results suggest that combining deep learning and knowledge from radiology reports in a hybrid framework can significantly enhance overall performance in the CXR finding detection.