Sequential 3D Reconstruction

Abstract:

3D reconstruction from photographs is a fundamental computer vision problem with applications in robotics, augmented reality, and scene interpretation. While deep learning has substantially improved reconstruction quality, managing sequential and temporal aspects remains challenging. This thesis investigates three complementary problems: predicting sequential assembly instructions, learning from streaming observations, and reconstructing dynamic scenes from monocular videos.

We first propose TreeSBA, a framework for generating brick-by-brick construction sequences from multi-view images. Our computationally efficient breadth-first search LEGO-Tree structure and class-agnostic tree-transformer architecture enable learning from synthetic pre-training and real-world image fine-tuning without assembly annotations. TreeSBA achieves 7.8% and 11.3% improvements in mean IoU on MNIST and ModelNet Construction datasets.

Second, we introduce UNIKD, an uncertainty-guided incremental learning approach for neural implicit representations. Unlike existing methods requiring simultaneous access to all training images, UNIKD enables continuous 3D model updates from sequential observations without catastrophic forgetting. Through uncertainty modeling and knowledge distillation between student-teacher networks, UNIKD improves F1 scores by 39.6% and 61.3% on ICL-NUIM and Replica datasets.

Third, we present 4D3R for pose-free novel view synthesis from monocular videos. Integrating transformer-based motion priors with SAM2 segmentation, our Motion-Aware Bundle Adjustment enables robust camera pose estimation. The efficient Motion-Aware Gaussian Splatting representation using sparse control points achieves 1.8dB PSNR improvement while reducing computational requirements by 5x.

These contributions advance sequential and temporal 3D understanding, providing efficient solutions for real-world scenarios where observations arrive gradually, scenes change dynamically, and direct supervision is limited.