Large-Scale Neural 3D Scene Reconstruction, Rendering, and Beyond

Abstract:

The ability to reconstruct and understand 3D scenes from 2D images is foundational for applications ranging from autonomous systems to cultural heritage preservation. While traditional methods based on handcrafted features and Multiview stereo struggle with scalability and robustness, neural rendering techniques like NeRF and 3D Gaussian Splatting (3DGS) have emerged as promising alternatives. However, critical challenges remain in building practical systems: (1) efficient processing of large-scale scenes, (2) robustness to imperfect camera poses, and (3) consistent reconstruction across distributed captures.

This thesis presents a series of advances that transform neural 3D reconstruction into a scalable and robust process through innovations at each stage of the pipeline. First, we address scalability with AdaSfM, a coarse-to-fine Structure from-Motion approach that enables distributed camera pose estimation by dynamically adapting to scene complexity and scale. Building on this foundation, DOGS introduces a distributed training framework for 3D Gaussian Splatting that achieves efficient large-scale reconstruction through novel consensus mechanisms, reducing computation and memory requirements by an order of magnitude compared to non-distributed approaches. To enhance robustness, we develop DBARF, which eliminates the dependency on accurate precomputed poses by jointly optimizing camera parameters and neural scene representation during NeRF training. This proves particularly valuable in challenging conditions like texture less regions where traditional SfM fails. For consistent reconstruction across distributed systems, DReg-NeRF automatically aligns neural radiance fields from different coordinate frames without manual intervention.

Together, these contributions form a comprehensive framework that pushes neural 3D reconstruction toward real-world applicability:

• Scalability and Efficiency: Distributed pose estimation (AdaSfM) and rendering (DOGS) for city-scale scenes.

• Robustness: Pose-free neural rendering (DBARF) for challenging environments.

• Consistency: Automatic alignment (DReg-NeRF) for unified reconstruction.

By simultaneously addressing efficiency, robustness, and consistency, this thesis not only advances the technical state-of-the-art but also enables practical deployment of neural 3D reconstruction in real-world scenarios where traditional approaches fall short. The proposed methods have been rigorously validated across diverse real-world datasets, demonstrating significant improvements in both accuracy and computational efficiency.