Comparative evaluation of YOLOv8, YOLOv11, and RT-DETR for automated microcarrier colonization assessment in...

Monitoring microcarrier colonization during mesenchymal stromal cell (MSC) expansion in bioreactors is essential for process control. Current manual quantification methods are time-consuming, subjective, and prone to inter-operator variability. This study presents a proof-of-concept evaluation of deep learning-based object detection for automated microcarrier colonization quantification in bioreactor cell cultures.

Three state-of-the-art architectures-YOLOv8, YOLOv11, and RT-DETR-were compared within a unified experimental framework. A dataset of 699 fluorescence microscopy images containing 46,982 annotated microcarriers (classified as colonized or non-colonized) was used for training and evaluation. Model selection followed a three-stage Successive Halving Algorithm (SHA) strategy to efficiently identify the best-performing architecture across 12 configurations without requiring high-end computational infrastructure. COCO-pretrained transfer learning was applied across all models.

YOLOv8-l emerged as the top-performing model, achieving mAP50-95 of 0.855 on validation and 0.787 on test, with a colonization estimation error (Mean Absolute Error) of 11.75%. The dataset size proved sufficient for CNN-based architectures, while also revealing the higher data demands of transformer-based detectors.

These results demonstrate that automated deep learning pipelines can reliably quantify microcarrier colonization, offering a practical alternative to manual assessment in cell therapy manufacturing. The findings also suggest that newer YOLO models do not necessarily yield improvements in domain-specific biomedical tasks.