Face-vid2vid and GANcraft
Abstract:
In this talk, I will present our face-vid2vid and GANcraft works. Face-vid2vid is a neural talking-head rendering engine we developed for the video-conferencing application. It learns a keypoint representation and decomposition in an unsupervised manner. The representation and decomposition allow us to extract a compact representation from a video frame on the sender side and reconstruct it faithfully on the receiver side. Empirically, we find it achieves 10x bandwidth saving compared to H.264. We also find it allows other applications such as character animation and face redirection. GANcraft is a neural rendering engine for Minecraft worlds. It learns to map the blocky Minecraft worlds to photorealistic real worlds. The key innovation is a way to marry GANs and NeRF. Through the pseudo-image supervision, we overcome the challenge of no corresponding real-world imagery for a Minecraft world.
Biodata:
I am a Distinguished Research Scientist and a Manager with NVIDIA Research. My research group focuses on deep generative models and their applications. We have created several high-impact research works in image synthesis. Several of them have enabled exciting new products, including NVIDIA Canvas and NVIDIA Maxine. I am always looking for talents to join our force. Please DM to my NVIDIA email if you are also passionate about generative models and their applications.
Prior to NIVIDA, I was a Principal Research Scientist with Mitsubishi Electric Research Laboratories (MERL). I received my Ph.D. degree from the University of Maryland, College Park, MD, USA, in 2012, advised by Prof. Rama Chellappa.