AFFECT ANALYSIS IN VIDEO

Abstract:

Affective computing is currently an active area of research, which is attracting an
increasing amount of attention. With the diffusion of ?affective computing? in many
application areas, ?affective video content analysis? is being extensively employed to help
computers discern the affect contained in videos. However, the relationship between the
syntactic content of the video, which is captured by low level features, and the expected
emotion elicited on humans remains unclear, while not much work has been done on the
evaluation of the intensity of discrete emotions.

In this thesis, we first propose a computational framework to build the representation
and modeling from the affective video content to the categorical emotional states, while
developing a computational measure for the intensity of ?categorical? emotional states.
Specifically, a sparse vector representation is proposed in this computational framework.
The ?intensity? of emotion can be represented by the values computed from the sparse
vector. Then, the modeling of affective content video addresses the problem of obtaining
the representative sparse vectors based on the low-level features extracted from video.
The results demonstrate that the proposed approach manages to represent and model
the affective video content based on the ?categorical emotional states? model, and the
obtained intensity time curve of the main emotion is in concurrence with the video
content. The second aim of this thesis is to examine the importance of the ?affect?
in the area of multimedia systems, by utilizing the sparse representation modeling in
applications. We therefore develop some useful applications towards this aim.

First, we propose an approach that employs affective analysis to automatically create
video presentations from home videos. Our novel method adaptively creates presentations
for family, acquaintances and outsiders based on three properties: emotional tone,
local main character and global main character. Experimental results show that our
method is very effective in video sharing and the users are satisfied with the videos
generated by our method.

Besides the adaptive presentation of home videos, this thesis also exploits the affective
analysis (facial expression cue), eye gaze data and previous emotional states to develop a
multi-modal approach combining for online estimating the subtle facial expression. It is
found that the performance of recognizing ?surprise? and ?neutral? emotions is improved
with the help of eye pupil information; namely, this result demonstrates that the fusion
of facial expression, pupillary size and previous emotional state is a promising strategy
for analyzing subtle expression.

Furthermore, this thesis also utilizes the affective analysis to propose a novel approach
to share home photos based on the aesthetic, affective and social features. This approach
allows one to generate a suitable subset of photos from the personal photo collection
for sharing with different social kinship groups. It can also be used to check whether
an individual photo is appropriate for sharing with a particular kinship group. Our
experiments demonstrate the utility of the proposed approach.

In view of the entire work in this thesis, our work is the first to evaluate the intensity
of emotions considering the ?categorical emotional states?; the first work to fuse the
facial expression, pupil size and previous emotional state to classify the subtle facial
expressions; and the first work to propose the concept of adaptive sharing of photos as
well. Based on affective modeling, in future, more interesting and useful applications
can be developed.