Generalizable Scene Understanding

Abstract:

Scene understanding is a fundamental research area in computer vision, with applications in autonomous driving, surveillance systems, AI assistants, and AI-generated content. While modern deep neural networks have achieved remarkable success under fully supervised settings, they often suffer from significant performance degradation when encountering unseen categories or new environments—scenarios that are prevalent in real-world applications. Therefore, developing robust and generalizable scene understanding techniques is crucial for ensuring model reliability in diverse settings. This thesis focuses on advancing deep learning-based algorithms for generalizable scene understanding in both 2D and 3D modalities, addressing key out-of-distribution challenges in object recognition, semantic segmentation, and object detection. The primary causes of performance degradation in these tasks stem from semantic shift and covariate shift between training and inference data. Semantic shift occurs when novel categories emerge that were not present during training, while covariate shift arises due to variations in data distributions, such as environmental changes or sensor differences. This thesis first investigates the semantic shift in 2D semantic segmentation and then explores the covariate shift in various scene understanding tasks in both 2D and 3D. By addressing these challenges, this thesis aims to improve the robustness and adaptability of scene understanding models, making them more effective in real-world scenarios where semantic and covariate shifts are inevitable.