Related Experiment Video
Updated: Aug 5, 2026
In vivo Structural Assessments of Ocular Disease in Rodent Models using Optical Coherence Tomography
Published on: July 24, 2020
Integration of Transformer-Based Architecture and Large Language Models for Optical Coherence Tomography Data
O V Konshina1, A D Pershin2, M K Kulyabin3
1Master Student, Educational and Research Center "Artificial Intelligence"; Ural Federal University named after the First President of Russia B.N. Yeltsin, Institute of Radioelectronics and Information Technologies - RTF, 32 Mira St., Ekaterinburg, 620062, Russia; Ophthalmologist; MIK Clinic, 21 Korablestroiteley St., Bldg. 1, Block B, Saint Petersburg, 199226, Russia.
Abstract:
The aim of the present study was to improve the differential diagnosis accuracy of retinal diseases combining a transformer model to classify the biomarkers on OCT images and the large language model DeepSeek-V3.
Materials And Methods:
Two datasets were collected and annotated: a training set (3288 central retinal OCT images annotated for 8 biomarkers) and a validation set (50 clinical cases from octcases.com). For biomarker classification, we compared ResNet, DenseNet, EfficientNet, and Vision Transformer (ViT-Tiny-Patch16-224) architectures. The ViT-Tiny model demonstrated the highest performance, its F1 macro - 0.84±0.03. An integration algorithm was developed to combine predicted biomarker labels with clinical history data via the DeepSeek-V3 LLM API.
Results:
Combining OCT biomarkers and the medical history significantly improved the diagnostic accuracy: Top-1 Accuracy - 78%, Top-3 Accuracy - 94%, MRR - 84%, representing 10-44% improvement over using either data type alone.
Conclusion:
The suggested approach enabled to automate the detection of biomarkers on OCT images and enhance the differential diagnosis accuracy of eye diseases, reducing the image interpretation time and supporting clinical decision-making.