On the Decoding of Human Pose

Abstract:

The field of deep learning based computer vision has experienced widespread growth in recent years where model-centric and data-centric AI become the mainstream. These paradigms focus on either models, loss functions, network parameters, or selection of data. However, different tasks in computer vision have different output formats; therefore, there exists a connecting layer from the network features to the output, which we call the "decoding'' layer. For example, if the network is required to classify 100 classes, there are two possible decodings, one with a 100-dimensional vector and the other with a number range from 0 to 99. It is obvious that the former decoding is much easier for the network to learn. In this thesis, we systematically study the decoding, whereas in the field of human pose estimation, where localize the keypoints in either 2D or 3D space. Specifically, we investigate the decoding of 2D/3D human pose estimation from three aspects: 1) the decoding of 2D pose, 2) the decoding of confidence estimation in 2D pose, and 3) the decoding of 3D pose (rotation).

The study of 2D pose decoding starts by analyzing two different decodings, i.e., detection- and integral regression-based, which can be distinguished by using argmax or soft-argmax function. The decoding of integral regression is learnable end-to-end, but has lower accuracy than detection. We uncover an induced bias from integral regression that results from combining the softmax and the expectation operation. This bias often forces the network to learn degenerately localized heatmaps, obscuring the keypoint's true underlying distribution and leads to lower accuracies. Training-wise, by investigating the gradients of integral regression, we show that the implicit guidance of integral regression to update the heatmap makes it slower to converge than detection. To counter the above two limitations, we propose Bias Compensated Integral Regression (BCIR), an integral regression-based decoding that compensates for the bias. BCIR also incorporates a Gaussian prior loss to speed up training and improve prediction accuracy.

Secondly, we investigate the decoding of 2D pose confidence, which should be aligned with the pose accuracy calibration-wise. Yet existing pose estimation methods tend to decode confidence with heuristics such as the maximum value of heatmaps. We show, through theoretical analysis and empirical verification, a calibration gap in current pose estimation frameworks induced by the decodings. Our derivations directly lead to closed-form adjustments in the confidence based on additionally inferred instance size and visibility. We go one step further and propose a Calibrated ConfidenceNet (CCNet) to explicitly learn network-specific adjustments {with a confidence prediction branch} to decode the 2D pose confidence. The proposed CCNet, as a lightweight post-hoc addition, improves the calibration of standard off-the-shelf pose estimation frameworks.

Thirdly, we focus on the decoding of 3D rotations, which is a common representation for 3D computer vision, including 3D human pose estimation. One form of representation -- rotation matrices -- is popular due to its continuity, especially for pose estimation tasks. The learning process usually incorporates orthogonalization to ensure orthonormal matrices. We treat the regression of rotation matrices from the perspective of multi-task learning as estimating orthonormal columns. Our work reveals, through gradient analysis, that common orthogonalization procedures based on the Gram-Schmidt process and singular value decomposition will cause gradient conflicts and slow down training efficiency. To this end, we advocate removing orthogonalization from the learning process and learning unorthogonalized `Pseudo' Rotation Matrices (PRoM), which leads to faster convergence and improved performance. By replacing the orthogonalization incorporated decoding with our proposed PRoM in various rotation-related tasks, we achieve state-of-the-art results on large-scale benchmarks for human pose estimation.

In summary, this thesis outlines three studies that explore the characteristics of current decoding of 2D pose, 2D pose confidence, and 3D pose. We demonstrate that the decoding layer is critical and needs both theoretical and empirical understandings. By improving the decoding, the method will be enhanced over all models and datasets.