Simplifying 3D Learning: Integrating Prior Knowledge and Decomposing Complexity
Abstract:
The field of 3D learning and modeling faces fundamental challenges: data scarcity and lack of supervision, difficulties in integrating prior knowledge effectively, and exponentially increasing computational demands for high-resolution outputs.
This thesis addresses these challenges through two complementary principles: the strategic integration of prior knowledge and the systematic decomposition of complex problems into tractable components. By leveraging domain-specific priors and breaking down intricate 3D learning tasks, we develop methods that achieve superior performance with reduced data requirements and computational overhead.
Our research makes three primary contributions that advance the state-of-the-art in 3D modeling:
First, we introduce a weakly-supervised keypoint-based framework for 3D pose transfer that overcomes the scarcity of paired training data. Our approach successfully disentangles pose and shape information from limited supervision, enabling cross-character pose transfer without requiring extensive datasets of different characters performing identical poses. This framework demonstrates how the use of keypoint priors can significantly reduce data requirements while maintaining high-quality results.
Second, we present the Human Gaussian Model (HGM), a diffusion-guided framework for generalizable 3D human modeling from single-view images. By integrating human anatomical priors through the SMPL-X model with advanced diffusion techniques, our approach enables high-quality human reconstruction that generalizes effectively to novel viewpoints and previously unseen subjects. This work showcases how domain-specific prior knowledge can dramatically improve generalization capabilities in few-shot scenarios.
Third, we develop MAR-3D, a Progressive Masked Auto-regressor for high-resolution 3D generation that decomposes complex joint distributions into tractable auto-regressive components within continuous space. This decomposition strategy enables the generation of high-quality 3D meshes while maintaining computational efficiency, demonstrating how systematic complexity reduction can make advanced 3D generation accessible with limited computing resources.
These contributions are unified by the overarching theme of simplifying 3D learning through principled approaches to prior integration and complexity management. Through extensive validation on diverse datasets and real-world scenarios, our methods demonstrate significant improvements in data efficiency, generalization capability, and computational tractability compared to existing approaches and offer practical 3D modeling solutions for the broader research community and industry applications.