PolypSAM-Open: Mitigating Automation Bias in AI-Assisted Colonoscopy via Open-Set Surgical Artifact Rejection

Background: Intelligent decision support systems for colonoscopy can fail when encountering out-of-distribution surgical instruments such as snares or biopsy forceps, producing false-positive polyp masks that may contribute to automation bias and reduce workflow reliability. This study aimed to develop and evaluate a parameter-efficient framework for open-set-aware polyp segmentation that can reject such anomalous inputs while preserving in-distribution segmentation performance. Methods: We propose PolypSAM-Open, which integrates a prototype-based Open-Set Learning (OSL) module with Low-Rank Adaptation (LoRA) in the MedSAM image encoder. The model was trained on Kvasir-SEG using an 85/15 split of authentic polyp images and synthetic high-frequency Gaussian noise to learn a rejection margin. Zero-shot out-of-distribution detection was evaluated on 590 unseen authentic surgical instruments from Kvasir-Instrument. Segmentation and detection performance were compared against a standard MedSAM-LoRA baseline. Results: Standard parameter-efficient fine-tuning yielded an OOD AUROC of 0.4263 on authentic surgical instruments. PolypSAM-Open improved zero-shot OOD AUROC to 0.9535 (p<0.001). Despite allocating 15% of training capacity to the synthetic-noise rejection margin, PolypSAM-Open maintained segmentation performance comparable to the standard fine-tuned baseline (Dice 0.9728 versus 0.9723). On ETIS-LaribPolypDB and CVC-ClinicDB, Dice scores were 0.9301 and 0.9386, respectively. The approach remained parameter-efficient, updating 4.48% of total parameters while adding negligible inference latency relative to the underlying MedSAM forward. Conclusions: Prototype-based open-set adaptation can substantially improve rejection of unseen surgical artifacts in AI-assisted colonoscopy while preserving high segmentation accuracy. These findings position PolypSAM-Open as a promising strategy for potentially safer decision-support segmentation in endoscopic workflows; prospective clinical validation remains necessary.