Towards Adversarially Robust Representations: Evaluation, Pre-Training, and Fine-Tuning
COM1 Level 2
SR6, COM1-02-03
Abstract:
Self-supervised pre-training on vast unlabelled datasets can yield generalizable representations, which have exhibited remarkable performance across diverse downstream tasks through fine-tuning. Nevertheless, the fine-tuned representations are not resistant to adversarial attacks in downstream tasks, which can be deceived into erroneous predictions. In this thesis, we advance the adversarial robustness of representations in terms of evaluation, pre-training, and fine-tuning, ultimately enabling safer and more reliable downstream applications.
Firstly, we demonstrate a prompt-based adversarial attack (PromptAttack) to evaluate the adversarial robustness of language representations adaptively. PromptAttack leverages the large language model to generate adversarial data that can successfully fool itself via an attack prompt. PromptAttack empirically unveils the severe adversarial vulnerabilities of representations in natural language processing contexts to such threats. The security risks caused by adversarial attacks necessitate robustifying representations, especially in safety-critical applications.
Secondly, we introduce two techniques for effectively and efficiently learning adversarially robust vision representations via self-supervised robust pre-training (RPT). Adversarial contrastive learning (ACL), which incorporates adversarial data with standard contrastive learning, is currently the most popular RPT method. However, the effectiveness and efficiency of RPT currently bottleneck the practicality of adversarially robust vision representations.
To enhance the effectiveness of RPT, we introduce an adversarial invariant regularization (AIR). We first built a causal theoretical framework to interpret ACL. Based on the causal understanding of ACL, AIR is derived to regulate and improve the ACL by enforcing the representations of natural data and adversarial data to be independent of the style factors. Empirical results validate that AIR can significantly enhance the adversarial robustness of fine-tuned representations in downstream tasks.
To improve the efficiency of RPT, we present a robustness-aware coreset selection (RCS) method. RCS does not require label information and searches for an informative unlabeled training subset that can maintain adversarial robustness with a theoretical guarantee. Empirically, RCS can significantly speed up ACL without sacrificing much adversarial robustness. For the first time, RCS enables the application of ACL to large-scale datasets like ImageNet-1K, significantly enhancing the scalability of RPT.
Lastly, we propose a hyperparameter-free and effective robust fine-tuning (RFT) framework to unleash the power of adversarially robust vision representations in downstream applications. Our proposed method introduces an auxiliary low-rank branch and an automated scheduler of key hyperparameters into the existing RFT framework, dubbed AutoLoRa. Empirically, by fine-tuning adversarially robust vision representations through AutoLoRa, we achieve the state-of-the-art adversarial robustness and standard generalization across various downstream datasets.