Towards Practical Continual Learning
Dr Lee Mong Li Janice, Professor, School of Computing
COM1 Level 2
SR7, COM1-02-07
Abstract:
Deep neural networks have demonstrated significant advances across diverse machine learning domains, often surpassing human performance in tasks involving vision, language, and speech. However, these models are typically trained for static settings, assuming fixed data distributions and known label spaces, which limits their ability to operate in evolving environments. Continual learning has emerged as a promising paradigm to address this limitation by enabling models to learn continuously from data streams without forgetting previously acquired knowledge.
While many methods have been proposed to mitigate catastrophic forgetting, the primary challenge in continual learning, several critical issues remain underexplored, limiting practical deployment in real-world applications. This thesis focuses on advancing continual learning toward practical applicability by addressing key challenges including knowledge transfer, multi-modal learning, robustness to novel data, and training efficiency.
We first propose a continual learning framework that promotes knowledge transfer across tasks by leveraging previously learned representations to facilitate learning of new classes. The framework comprises a dynamic architecture with expanding representations to preserve previously learned features while accommodating new ones, and employs a training strategy alternating between two objectives to balance the performance between new and old classes. Beyond improved accuracy and reduced forgetting, qualitative analysis shows evidence of effective knowledge transfer, where newly learned features refine earlier class representations.
Next, we extend continual learning to multi-modal settings, addressing the challenge of retaining and leveraging cross-modal relationships learned over time. We introduce a multi-modal continual learning framework based on a novel cross-modality adapter with a mixture-of-experts structure, enabling effective integration of multi-modal information across tasks. To preserve prior knowledge and support robust multi-modal representation learning, we also incorporate a representation alignment loss and regularise inter-representation relationships. Extensive experiments show that our approach consistently outperforms baseline methods, achieving higher accuracy and reduced forgetting.
We further address continual learning in open-world environments, where a learner must both adapt continuously and identify instances it has yet to learn. To this end, we develop a multi-modal online continual learning framework with a novelty estimation mechanism based on dynamically determined, class-specific uncertainty thresholds. These uncertainty estimates support both novel class detection and adaptive balancing between knowledge acquisition and retention. Our approach consistently demonstrates superior performance in classifying known classes, detecting novel samples, and mitigating forgetting.
Finally, we tackle efficient continual adaptation of foundation models, focusing on both parameter and data efficiency while preserving their strong zero-shot capabilities on unseen tasks. We introduce a lightweight dynamic prompting mechanism that generates instance-specific prompts for adaptation, together with a novel augmentation strategy that creates semantically meaningful mixed images paired with mixture-aware text embeddings. With substantially fewer trainable parameters, our approach achieves improved adaptation performance, reduced forgetting, and strong zero-shot transfer, yielding consistent gains even under partial supervision.
Overall, this thesis moves beyond a narrow focus on avoiding forgetting and advances continual learning toward practical deployment, emphasizing adaptability, generalisability, robustness, and efficiency.