From Scarcity to Synthesis: Continual Learning Integrates Supervised and Unsupervised CT Image Recovery Models

A strong deep learning model typically relies on a large dataset that exposes it to a variety of scenarios for learning. However, data scarcity remains a long-standing and widely recognized challenge in the field of CT image recovery. Existing datasets vary in scanner types, noise levels, and training paradigms, resulting in fragmentation. Continual learning offers a promising solution, enabling sequential training across multiple distinct datasets, thereby alleviating data scarcity. However, such sequential training introduces catastrophic forgetting, where previously learned knowledge is lost. While continual learning traditionally aims to mitigate this issue, our work shows that lower forgetting in CT image recovery is associated with better retention and stronger adaptability to unseen tasks. Through extensive empirical analysis, we report two critical observations: (1) when applying continual learning to CT image recovery, the degree of catastrophic forgetting of knowledge from previous datasets inversely correlates with model performance on newly introduced datasets; and (2) different datasets with different training paradigms exhibit shared, transferable representations and knowledge. Motivated by these findings, we propose a task-agnostic continual learning framework that unifies these regimes by training on a sequence of supervised and unsupervised tasks with selective experience replay. This method balances curated supervised and unsupervised exemplars and employs dual-weight knowledge distillation to regularize new task learning while strongly preserving past predictions. Our method outperforms state-of-the-art continual learning baselines and approaches the joint training upper bound, achieving reduced forgetting and knowledge transfer on sequential low-dose CT denoising tasks. These results support continual learning as a practical solution for CT recovery, enabling cross-paradigm knowledge sharing, strong reconstruction fidelity, and sustained performance across different datasets.