A Theory of Continual Learning with Gradient Descent for Neural Networks
COM3 Level 2
SR21, COM3 02-60
Abstract:
Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting the earlier ones, is a central goal of artificial intelligence. To shed light on its underlying mechanisms, we analyze the limitations of continual learning in a tractable yet representative setting. In particular, we study one-hidden-layer quadratic neural networks trained by gradient descent on an XOR cluster dataset with Gaussian noise, where different tasks correspond to different clusters with orthogonal means. Our results obtain bounds on the rate of forgetting during train and test-time in terms of the number of iterations, the sample size, the number of tasks, and the hidden-layer size. Our results reveal interesting phenomena on the role of different problem parameters in the rate of forgetting. Numerical experiments across diverse setups confirm our results, demonstrating their validity beyond the analyzed settings.
Joint work with Hossein Taheri.
Bio:
Arya Mazumdar is a Hal??c??o??lu Data Science Institute Endowed Chair Professor in AI at University of California San Diego, and the Deputy Director and Associate Director of Research of the NSF-funded AI Institute for Learning-Enabled Optimization at Scale (TILOS) Institute. He is also a Co-PI and UCSD Site Lead of NSF-TRIPODS Phase II institute, EnCORE. His research areas cover algorithmic and statistical aspects of machine learning, error correcting codes, optimization, and signal processing. Currently, his main focus is machine learning, specifically computational aspects of statistical estimation/inference and distributed learning. Previously, his work in information and coding theory led to major advancements in the problem of local data recovery and sparse signal recovery. Awards received by Prof. Mazumdar include the 2011 ECE Distinguished Dissertation Award and 2025 ECE Distinguished Alumni Award (both from Univ. of Maryland), EURASIP Best Paper Award, IEEE ISIT Jack Keil Wolf paper award, and a 2015 NSF CAREER Award.