A collaborative hybrid network with aggregated attention for pulmonary nodule detection

Lung cancer remains one of the most lethal malignancies worldwide, underscoring the critical importance of early detection for improving patient survival. Current imaging‑based methods for pulmonary nodule (PN) detection are often limited by low efficiency and suboptimal accuracy, falling short of the demands for precise clinical diagnosis. The baseline YOLOv8 model, although efficient, is prone to missed detections due to non-maximum suppression (NMS)‑induced suppression of small nodules and insufficient feature extraction for morphologically complex lesions. To address these clinical challenges, this study aimed to optimize YOLOv8 by integrating a real-time detection transformer (RT-DETR) decoder head and an aggregated attention mechanism, thereby improving detection sensitivity and accuracy for PNs of varying sizes and shapes.

The proposed model, termed YOLOv8‑RTA, integrates an RT‑DETR‑based transformer decoder head into YOLOv8, replacing the original detection head to mitigate NMS‑induced suppression of true boxes, and introduces an aggregated attention mechanism within the backbone to expand the effective receptive field and enhance deep feature discriminability. Training and evaluation were performed on the Tianchi dataset, with five‑fold cross‑validation and an external test on 50 hospital Picture Archiving and Communication System (PACS) computed tomography (CT) scans.

YOLOv8‑RTA achieved precision of 93.28%, recall of 92.4%, and mean average precision at 50% intersection over union (mAP50) of 94.98% on the Tianchi dataset. Compared to the baseline YOLOv8 model, which achieved an mAP50 of 90.6%, the proposed method improved the detection accuracy by 4.38 percentage points. It also outperformed RT‑DETR‑R18, which attained an mAP50 of 93.51%, by a margin of 1.47 percentage points. The model contains only 6.24 million parameters and runs at 12.2 giga floating point operations per second (GFLOPs). In five‑fold cross‑validation, the mAP50 ranged from 75.4% to 99.3%, with a mean value of 91.5%. On the external PACS dataset, the model detected all 50 nodules, achieving a mean confidence score of 0.86 for nodules smaller than 5 mm.

This study proposes YOLOv8‑RTA, a collaborative hybrid network that combines an RT‑DETR decoder head and an aggregated attention module to enhance multi‑scale feature extraction and discriminative representation for PN detection. On the Tianchi dataset, the model achieved improved performance over baseline YOLOv8 and RT‑DETR‑R18, with gains of 4.38 and 1.47 percentage points in mAP50, respectively, while maintaining a lightweight architecture of only 6.24 million parameters. A preliminary test on 50 real‑world PACS CT images suggests potential clinical feasibility, with all nodules successfully detected in this limited cohort. These findings indicate that YOLOv8‑RTA offers a favorable accuracy-efficiency trade‑off for PN screening. Future work will focus on larger multi‑center validation, patient‑level cross‑validation, and extension to three-dimensional (3D) context modelling.