Advancing Domain Adaptive Pose Estimation For Evolving Challenges
COM1 Level 3
MR1, COM1-03-19
Abstract:
Pose estimation from RGB images aims to predict hand or human body joint locations in 2D or 3D, with applications ranging from animation, robotics, and AR/VR. The scarcity of annotated data and labelling difficulties often lead to reliance on synthetic or indoor datasets, resulting in a domain gap with real-world data in appearance and pose diversity. Bridging this gap requires domain adaptation methods to enable models trained on labelled data to perform effectively on unseen real-world data without 2D or 3D annotations. This thesis aims to build a comprehensive understanding of domain adaptive pose estimation and develop practical solutions that address progressively constrained real-world scenarios from UDA (Unsupervised Domain Adaptation) to TTA (Test-time Adaptation) and OTTA (Online Test-time Adaptation).
First, in UDA, pre-training with synthetic data typically involves using only labelled RGB images, overlooking other easily generated modalities during rendering, such as depth maps. Depth maps can provide additional geometric cues that complement RGB images by capturing shared task-relevant information of the underlying structure. The shared information is useful for enhancing hand pose estimation in unseen data. To exploit this, we design a network architecture that integrates depth maps to enhance the RGB-based hand pose estimation via a multi-level alignment. By reducing the RGB encoder’s sensitivity to non-informative cues like backgrounds or irrelevant textures, this approach significantly improves model generalization to unseen data.
Second, fine-tuning on unlabelled datasets often employs pseudo-labelling, but its effectiveness is limited due to inherent contradictions. Only sufficiently accurate pseudo-labels enhance learning, yet generating high-quality pseudo-labels suggests the model is already capable of robust pose estimation. The cross-domain nature of the problem hinders the generation of reliable pseudo-labels. To address this, we propose leveraging the unlabelled RGB data to refine keypoint predictions through a reconstruction loss derived from the estimated keypoints. This approach improves confident but incorrect keypoints by aligning with 2D evidence while preserving the quality of accurate ones.
Third, we focus on TTA where the model adapts during inference without access to labelled training data. TTA methods often rely on a 2D projection loss, which inherently suffers from depth ambiguity and struggles when 2D keypoints are missing. While temporal information can help mitigate the above problems, it tends to produce over-smoothed estimates that lose essential semantic details, although it aligns well with 2D observations. To tackle this, we utilize semantics-aware motion as a prior regularization and additional guidance. This approach not only reduces the ambiguous 3D solution space when 2D keypoints are available but also generates useful 2D projected poses to fill in missing information when keypoints are absent.
Finally, we focus on OTTA scenario, where the model updates only on each arriving sample and gives predictions without a forecast of future frames, making it more aligned with practical requirements. This setting intensifies the challenges associated with the estimated 2D poses. A fundamental trade-off emerges: while estimated 2D poses provide valuable cues for refining 3D poses, their noise can cause the model to break down in OTTA, where only a limited number of frames are available for adaptation at a time. To address this, we propose a new training pipeline to mitigate error accumulation across videos and an early-stop strategy for each sample adaptation. These improvements help limit error propagation while preserving useful information from estimated 2D poses.