Identity Crisis: When Face Recognition Meets Twins and Privacy

Abstract:

Face recognition has received significant attention during the past several decades due to its wide applications, such as access control and law enforcement. Most works in this area focus on how to improve recognition accuracy for general population, while few works have studied how to recognize identical twins and how to smartly protect privacy. In fact, these two problems are very important to the wide usage of face recognition. Overlooking either of them may cause severe identity crisis. For identical twins, they share the same DNA code, thus their appearances are quite similar. Conventional face recognition systems which utilize facial appearance may be not suitable for identical twins anymore. For privacy protection, existing works only concern the protection of identity, while destroying other facial attributes, such gender and race. These attributes, however, may be beneficial for many intelligent visual analyses systems. In this thesis, we try to address those two problems by 1) utilizing behavioral biometrics (facial motions and voice) and ear biometric to distinguish between identical twins; 2) proposing SemanticFace which can independently control the amount of identity alteration while keeping unchanged other facial attributes to protect privacy without thwarting facial analyses systems.

For identical twins, we started with verifying the difficulty in appearance-based approaches through experiments on a 39 pairs of identical twins database collected by ourselves. Motivated by ``twins may look alike, but behave differently", we then proposed to utilize 1) skin deformation pattern from expression dynamics; 2) temporal trajectory pattern (talking profile) from head translation, pupil movement, pose change and mouth/eye open-close magnitude change during free conversation; 3) spectral pattern from voice, to distinguish between identical twins. The skin deformation pattern employed the Right-Cauthy tensor to describe the elastic property of skin which may provide identity evidence. Temporal trajectory pattern attempted to detect abnormal local motions and used it as biometric. Spectral pattern modeled spectral probability distribution of each subject to extract identity-specific signature. The experiments on our twin database demonstrated the effectiveness of these behavioral biometrics. We further used Siftflow registration to extract the shape and appearance information from ear images as ear biometric. The results showed that ears are identical between twins and can be a good candidate for twin recognition as well.

For privacy protection, we applied a subspace decomposition technique onto our face encoding scheme, effectively decoupling facial attributes such as gender, race, age and identity one from another into orthogonal subspaces. Consequently we can independently control these attributes during synthesis by altering their corresponding coefficients. Our method is thus useful for nuanced face de-identification, in which only facial identity is altered, but others, such gender, race and age, are retained. These altered face images protect identity privacy, and yet allow other facial analyses to proceed unimpeded. Our proposal also permits privacy to be applied not just to identity, but also to other facial attributes as well. Furthermore, privacy-protection mechanisms, such as k-anonymity, L-diversity, and t-closeness, may be readily incorporated into our method. Extensive experiments with a commercial facial analysis software show that our alteration method is indeed effective.