Machine Unlearning in Graphs, Recommendations, and Language Models
COM1 Level 3
MR1, COM1-03-19
Abstract:
Selective forgetting in machine learning has emerged as a transformative paradigm for addressing critical challenges related to data privacy, ethical compliance, and adaptability in dynamic environments. The rapid advancements in this field have intensified concerns over these issues, necessitating the development of machine unlearning—techniques that enable models to selectively forget specific information. By ensuring compliance with regulations like the "right to be forgotten," facilitating model adaptation, and reducing computational overhead, machine unlearning paves the way for responsible and efficient AI systems. This thesis explores novel unlearning methods across three interconnected domains: graph neural networks (GNNs), multi-modal recommender systems (MMRS), and large language models (LLMs).
For graph unlearning, we propose D2DGN, a knowledge distillation-based approach designed to delete elements of graph while preserving the integrity of the retained graph structure. By partitioning graph knowledge into retain and forget sets and using response-based and feature-based distillation, D2DGN achieves efficient unlearning without repetitive retraining. Experimental results on real-world datasets demonstrate significant improvements in unlearning performance, with up to 43.1% higher AUC in edge and node unlearning tasks compared to existing methods.
In the domain of multi-modal recommendation, we introduce MMRECUN, the first method to unlearn item data in MMRS. Leveraging a novel Reverse Bayesian Personalized Ranking (BPR) objective, MMRECUN diminishes the influence of user-item interactions within the forget set while reinforcing interactions in the retain set. This enables effective unlearning of outdated preferences, revoked licenses, or legally mandated removals. Experiments on benchmark datasets show recall improvements of up to 49.85% over baseline methods and computational speed gains of up to 1.3× compared to retraining.
For unlearning in LLMs, we propose UNSTAR, a framework that introduces the novel concept of anti-samples. These are generated using misleading rationales to reverse learned associations, enabling fine-grained targeted unlearning. Unlike prior approaches, UNSTAR selectively removes specific knowledge without degrading related information, offering a precise and efficient unlearning strategy. This approach has significant implications for privacy-preserving AI and adaptive model modification.
To complement these methods, we introduce a black-box evaluation framework designed to rigorously assess unlearning vulnerabilities. Our framework combines adversarial prompt generation, targeted attacks to probe unlearned content. Initial results reveal that current unlearning techniques often fail to ensure true knowledge erasure, instead suppressing information that remains accessible through adversarial probing.
All four contributions form a comprehensive framework for addressing unlearning challenges across diverse machine learning paradigms. By improving performance, efficiency, and granularity, this thesis lays the foundation for more adaptive, responsible, and privacy-compliant AI systems, advancing the state-of-the-art in machine unlearning.