Robust and Efficient Novel View Synthesis for Real-World 3D Scenes
COM1 Level 2
SR9, COM1-02-09
Abstract:
Novel view synthesis aims to produce photorealistic images of 3D scenes from limited 2D observations, enabling applications such as virtual tours, 3D content creation, robotic simulation, and digital twins. Despite recent advances, real-world deployment remains challenging due to limitations in robustness, scalability, and efficiency. This seminar presents a series of methods addressing these challenges across two key stages: reconstructing 3D representations and rendering novel views.
For reconstruction, we introduce a surface-aware neural radiance field framework with an improved reflection model to handle dynamic scenes with specular surfaces, where visual consistency across viewpoints is difficult to establish. We further present an efficient approach based on implicit multi-view guidance that supports on-the-fly training and rendering for real-time and streaming applications.
For rendering, we introduce a multi-scale 3D Gaussian representation that mitigates aliasing while maintaining speed and visual quality across varying viewing distances and image resolutions. Coarse-level Gaussians emerge from fine-grained primitives and are selectively activated to support scalable rendering at different levels of detail. To overcome the limitations of semi-transparent representations, we also propose an end-to-end method for reconstructing scenes with fully opaque triangles. By combining multi-layer occlusion modeling, image-space approximations, and tailored density control, this method enables direct optimization without a two-stage distillation process. It achieves rendering speeds exceeding 1,000 FPS and integrates with existing graphics engines to support lighting, materials, and special effects without custom shaders. Together, these contributions advance robust and efficient novel view synthesis and facilitate its practical use in complex real-world environments.