Sliding Window Optimal Transport for Open World Artifact Detection in Histopathology

Histological images are frequently impaired by local artifacts from scanner malfunctions or iatrogenic processes - caused by preparation - impacting the performance of Deep Learning models. Models often struggle with the slightest out-of-distribution shifts, resulting in compromised performance. Detecting artifacts and failure modes of the models is crucial to ensure open-world applicability to whole slide images for tasks like segmentation or diagnosis. We introduce a novel technique for out-of-distribution detection within whole slide images, compatible with any segmentation or classification model. Our approach tiles multi-layer features into sliding window patches and leverages optimal transport to align them with recognized in-distribution samples. We average the optimal transport costs over tiles and layers to detect out-of-distribution samples. Notably, our method excels in identifying failure modes that would harm downstream performance, surpassing contemporary out-of-distribution detection techniques. We evaluate our method for both natural and synthetic artifacts, considering distribution shifts of various sizes and types. The results confirm that our technique outperforms alternative methods for artifact detection. We assess our method components and the ability to negate the impact of artifacts on the downstream tasks. Finally, we demonstrate that our method can mitigate the risk of performance drops in downstream tasks, enhancing reliability by up to 77%. In testing 7 annotated whole slide images with natural artifacts, our method boosted the Dice score by 68%, highlighting its real open-world utility.