S4M: 4-points to segment anything

The Segment Anything Model (SAM) promises to ease the annotation bottleneck in medical segmentation, but overlapping anatomy and blurred boundaries make its point prompts ambiguous, leading to cycles of manual refinement to achieve precise masks. Better prompting strategies are needed.

To capture spatially meaningful cues at minimal annotation cost, we propose a structured prompting strategy using 4-points as a compact instance-level shape description. We study two 4-point variants: extreme points and the proposed major/minor axis endpoints, inspired by ultrasound measurement practice. Yet SAM cannot fully exploit such structured prompts, as it treats all points identically and lacks geometry-aware reasoning. To address this, we introduce S4M (4-points to Segment Anything), which augments SAM to interpret 4-points as relational cues rather than isolated clicks. S4M expands the prompt space with role-specific embeddings, disentangling each point's semantic and spatial role. An auxiliary "Canvas" pretext task further strengthens prompt representation learning by sketching coarse masks directly from prompts, without visual input, fostering geometry-aware reasoning from minimal interaction.

Across eight datasets in ultrasound and surgical endoscopy, S4M improves segmentation by +3.42 mIoU over a strong SAM baseline at equal prompt budget. An annotation study with three clinicians further demonstrate that major/minor prompts allow faster and practical annotation.

S4M increases performance, reduces annotation effort, and builds on conventions already embedded in clinical workflows. By aligning prompting with clinical practice, it removes a barrier to adoption and enables more scalable dataset development in medical imaging. The code is available at https://github.com/CAMMA-public/S4M .