Abstract
Foundational models are trained on extensive datasets to capture the general
trends of a domain. However, in medical imaging, the scarcity of data makes
pre-training for every domain, modality, or task challenging. Continual
learning offers a solution by fine-tuning a model sequentially on different
domains or tasks, enabling it to integrate new knowledge without requiring
large datasets for each training phase. In this paper, we propose UNIfied
CONtinual Learning for Medical Foundational Models (UNICON), a framework that
enables the seamless adaptation of foundation models to diverse domains, tasks,
and modalities. Unlike conventional adaptation methods that treat these changes
in isolation, UNICON provides a unified, perpetually expandable framework.
Through careful integration, we show that foundation models can dynamically
expand across imaging modalities, anatomical regions, and clinical objectives
without catastrophic forgetting or task interference. Empirically, we validate
our approach by adapting a chest CT foundation model initially trained for
classification to a prognosis and segmentation task. Our results show improved
performance across both additional tasks. Furthermore, we continually
incorporated PET scans and achieved a 5\% improvement in Dice score compared to
respective baselines. These findings establish that foundation models are not
inherently constrained to their initial training scope but can evolve, paving
the way toward generalist AI models for medical imaging.
Abstract
Continual learning (CL) is crucial for the adaptation of neural network
models to new environments. Although outperforming weight-space regularisation
approaches, the functional regularisation-based CL methods suffer from high
computational costs and large linear approximation errors. In this work, we
present a new functional regularisation CL framework, called MCFRCL, which
approximates model prediction distributions by Monte Carlo (MC) sampling.
Moreover, three continuous distributions are leveraged to capture the
statistical characteristics of the MC samples via moment-based methods.
Additionally, both the Wasserstein distance and the Kullback-Leibler (KL)
distance are employed to construct the regularisation function. The proposed
MCFRCL is evaluated against multiple benchmark methods on the MNIST and CIFAR
datasets, with simulation results highlighting its effectiveness in both
prediction accuracy and training efficiency.