A labeled ophthalmic ultrasound dataset with medical report generation based on cross-modal deep learning

Ultrasound imaging reveals eye morphology and aids in diagnosing and treating eye diseases. However, interpreting diagnostic reports requires specialized physicians. We present a labeled ophthalmic dataset for the precise analysis and the automated exploration of medical images along with their associated reports. It collects three modal data, including the ultrasound images, blood flow information and examination reports from 1,0361 patients at an ophthalmology hospital in Shenyang, China, during the year 2016 to 2020, in which the patient information is de-identified for privacy protection. To the best of our knowledge, it is the only ophthalmic dataset that contains the three modal information simultaneously. It incrementally consists of 2,2173 images with the corresponding free-text reports, which describe 10 typical imaging findings of intraocular diseases and the corresponding anatomical locations. Each image shows three kinds of blood flow indices at three specific arteries, i.e., nine parameter values to describe the spectral characteristics of blood flow distribution. The reports were written by ophthalmologists during the clinical care. In addition, the knowledge fusion cross modal network (KFCMN) is proposed to generate report according to the proposed dataset. The experimental results demonstrate that our dataset is suitable for training supervised models concerning cross-modal medical data.