The Space Between The Images -- Visual Learning From Relations
Abstract:
In understanding or generating images or videos, visual relations play a fundamental role, reflecting basic principles that underlie the physical world. These can range from symmetries and repetitions, to groupings and compositional structures, to invariance and equivariance under various transformations, to cycle consistency. Today, almost all supervision we provide to our models is first-order and value-driven, such as specifying desired color or semantic class for pixels, expected depth for 3D points, etc. Yet the most important visual structure is encoded in binary or multiway relations between these values --- reflecting compositional scene hierarchies or respect for geometric or physical laws. In this talk we examine a number of ways that second-order, relational supervision can be provided, from being baked into the model to contrastive learning, to consistency losses for RL. We show that relation-awareness can vastly reduce the amount of training data needed and lead to superior performance across multiple applications, including classification, segmentation, reconstruction, and VQA.
Bio:
Leonidas Guibas is the Paul Pigott Professor of Computer Science at Stanford University and a Principal Scientist at Google Deep Mind.. He has worked in numerous areas of computer science, such as geometric algorithms, 3D computer vision and geometric deep learning, computer graphics, robotics, discrete mathematics, and biocomputation. Dr. Guibas has been elected to the US National Academy of Engineering, the US National Academy of Sciences, the American Academy of Arts and Sciences, and the Siggraph Academy, He is is an ACM Fellow, an IEEE Fellow, and has won the ACM-AAAI Allen Newell Award, the ICCV Helmholtz prize, and Siggraph's Steven Anson Coons award.