Multi-modality multi-label ocular abnormalities detection with transformer-based semantic dictionary learning

Blindness is preventable by early detection of ocular abnormalities. Computer-aided diagnosis for ocular abnormalities is built by analyzing retinal imaging modalities, for instance, Color Fundus Photography (CFP). This research aims to propose a multi-label detection of 28 ocular abnormalities consisting of frequent and rare abnormalities from a single CFP by using transformer-based semantic dictionary learning. Rare labels are usually ignored because of a lack of features. We tackle this condition by adding the co-occurrence dependency factor to the model from the linguistic features of the labels. The model learns the relation between spatial features and linguistic features represented as a semantic dictionary. The proposed method treats the semantic dictionary as one of the main important parts of the model. It acts as the query while the spatial features are the key and value. The experiments are conducted on the RFMiD dataset. The results show that the proposed method achieves the top 30% in Evaluation Set on the RFMiD dataset challenge. It also shows that treating the semantic dictionary as one of the strong factors in model detection increases the performance when compared with the method that treats the semantic dictionary as a weak factor.