Data Efficient 3D Indoor Scene Understanding

Abstract:

3D indoor scene understanding endows the AI agents to understand the surrounding environment, which is widely used in robotics, augmented reality and health-care systems, etc. Particularly, we focus on two fundamental tasks: 3D semantic segmentation for point-level and 3D object detection for instancelevel scene understanding, respectively. The goal of 3D semantic segmentation is to predict category of each point in the 3D scenes typically represented by the point cloud while 3D object detection aims to predict the 3D bounding boxes of objects along with their respective classes.

While deep learning methods achieve impressive performance, they typically rely on 3D input for spatial reasoning and require large 3D training datasets to learn effectively, which can be hard to prepare. To this end, we study data efficient learning for 3D semantic segmentation and 3D object detection, that can learn from few 3D training data or using only 2D images as input.

To address learning from limited 3D training data, we conduct in-depth research on few-shot point cloud based 3D semantic segmentation. Although few-shot 3D semantic segmentation (FS-3DSeg) shows better generalization to new classes with only few training samples than fully-supervised methods, it assumes the point cloud in the support set are correctly labeled, which can be violated in real-world applications. Therefore, we explore robust few-shot 3D semantic segmentation to improve model’s robustness against the common instance-level annotation mistake in the support set. Our designed component-level clean noise separation (CCNS) and multi-scale degree-based noise suppression (MDNS) show significant improvement over the vanilla few-shot 3D semantic segmentor.

Another drawback of FS-3DSeg is that it only segments novel classes while ignoring the base classes learnt before. To make FS-3DSeg model maintain the knowledge of base classes, we then study generalized few-shot 3D semantic segmentation. We propose geometric words to represent diverse basic geometric structures that are shared across different classes. Based on geometric words, we design geometric-aware semantic representation (GSR) and geometric-guided classifier re-weighting (GCR) to enable model to maintain the knowledge of base classes and also effectively learn new classes with few annotated samples.

Although above methods alleviate 3D training data hungry problem, their input still need to be point cloud, which requires the expensive depth sensors to capture. To bypass the need of 3D data for the input, we study image-based 3D scene understanding through 3D object detection, which predicts 3D bounding boxes of objects in the scene from only posed RGB images. We introduce MVSDet to efficiently extract geometry information from the given 2D images for 3D object detection. Particularly, we propose a probabilistic sampling and soft weighting mechanism to predict the depth locations for every pixel without sampling many depth planes. We also novelly utilize pixel-aligned Gaussian Splatting as a lightweight regularizer to the depth prediction and improve detection performance.