Controllable Manipulation of Facial Images

Abstract:

Facial image synthesis has been an interesting and challenging problem for both computer graphics and computer vision. Its main objective is to generate photo-realistic facial images conditioned on certain semantic attributes. Over the last few decades, facial image synthesis techniques have been widely employed in various areas, such as film industry, computer game industry, and robust face recognition systems etc.

In literature, a vast variety of algorithms have been proposed for synthesizing facial images with different attributes, particularly as pose, illumination, expression and other facial appearance attributes such as hair color and accessories etc. The essential identity attribute of facial images is often overlooked. Meanwhile, among the existing algorithms for facial image synthesis, only a small portion target the task of synthesizing facial images with multiple attributes. To address this task, it is often needed to disentangle multiple face attributes and facial identity, which poses a great challenge especially when only partially-attributed disjoint face datasets are available.

In this thesis, we aim to achieve two objectives: (1) synthesizing controllable virtual identities for facial images in order to address the face template protection problem; (2) learning to disentangle and synthesize multiple attributes for facial images from partially-attributed disjoint datasets.

The first objective is motivated by the problem of private information leakage at the exposure of face templates in practical face recognition systems. To address this problem, we propose to re-assign a virtual identity to users by manipulating their identity feature representations. The virtual identities are carefully devised so that the new identities of users are still discriminative to each other. Evaluations on multiple scenarios show that our method is able to protect face templates without decreasing the face recognition performance.

The second objective fills in the gap of synthesizing facial images with multiple target attributes. In particular, it studies the problem of disentangling and synthesizing multiple-attributed facial images from several partially-attributed datasets. We propose a method that takes the advantage of generative adversarial network (GAN) architecture to synthesize photo-realistic facial images. In order to overcome the challenge by partially-attributed datasets, we employ a small auxiliary fully-attributed dataset to assist the learning of the facial image generator. Evaluations show that disentangled and discriminative attribute feature spaces are learned and photo-realistic facial images are synthesized.