Hierarchical visual prompt disentanglement for continual rare disease diagnosis

The automatic diagnosis of rare diseases via few-shot class-incremental learning (FSCIL) is critically hindered by severe data scarcity and the prevailing treatment of disease labels as a flat, unstructured space. This oversimplification neglects the rich taxonomic structures inherent in medical ontologies, leading to poor generalization on fine-grained subtypes. To bridge this gap, we propose Hierarchical Visual Prompt Disentanglement (HVPD), a hierarchy-guided framework for continual rare disease diagnosis. Unlike methods that use textual cues for flat semantic alignment, HVPD uses superclass and subclass category names to elicit visual-attribute semantics from a medical large language model (LLM) and converts the LLM's pre-decoding representations into hierarchical semantic anchors that guide image-conditioned visual prompt generation. We design a hierarchy-aware disentanglement mechanism that synthesizes instance-adaptive visual prompts while reducing the overlap between superclass-common patterns and subclass-specific diagnostic cues through an orthogonality-based cross-level constraint. This design alleviates the stability-plasticity dilemma by injecting structured medical priors into the visual feature space. Extensive experiments on the SD-198 (dermatology) and RFMiD (ophthalmology) datasets demonstrate that HVPD consistently improves over existing FSCIL approaches, delivering stronger fine-grained discrimination and knowledge retention under extreme few-shot regimes.