Towards Improving Generalization of Deep Neural Networks: Data, Parameter Optimization, and Deployment
COM3 Level 2
SR21, COM3 02-60
Abstract:
Deep neural networks have achieved impressive performance across numerous domains, but their ability to generalize to unseen data remains a fundamental challenge. In real-world applications, achieving consistent and reliable performance on unseen data is essential for deploying neural networks successfully. This generalization capability is crucial for building trustworthy AI systems. In this thesis, I will follow the traditional pipeline of neural network training and deployment, which typically involves three sequential stages: training data preparation, parameter optimization, and model deployment. I will examine how to improve generalization ability at each of these stages.
From the training data perspective, I propose ConAdv, which leverages adversarial training to collect adversarial examples that smooth the decision boundary, ultimately enhancing generalization ability.
From the parameter optimization perspective, I propose LookSAM, which utilizes parameter perturbation techniques such as sharpness-aware minimization (SAM) to guide the model toward convergence in flat regions, thereby improving generalization.
From the model deployment perspective, I propose SeedLoRA. Before deployment, multiple fine-tuned variants of the same pre-trained model (initialized with different random seeds) are merged in weight space to obtain a more robust final model that outperforms any individual run.
In summary, these three approaches demonstrate that generalization can be systematically improved throughout the entire neural network training pipeline, from data preparation to final deployment, providing practical solutions for building more robust neural networks in real-world scenarios.