Social Interaction Analysis Using A MultiSensor Approach

Abstract:
Humans are by nature social animals, and the interaction between humans is an integral feature of human societies. Social interactions play an important role in our daily lives: people organize themselves in groups to share views, opinions, as well as thoughts. However, as the availability of large-scale digitized information on social phenomena becomes prevalent, it is beyond the scope of practicality to analyze the big data computational assistance. Also, recent developments in sensor technology, such as the emergence of new sensors, advanced processing techniques, and improved processing hardware, provide an opportunity to improve the techniques for analyzing interactions by making use of more sensors in terms of both modality and quantity.

This thesis focuses on the analysis of social interactions from the social signal perspective in the multi-sensor setting. The thesis starts with our first work, in which we propose an extended F-formation system for robust interaction and interactant detection in a generic ambient sensor environment. The results on interaction center detection and interactant detection show improvement compared to the rule-based interaction detection method. Building upon this work, we study the spatial structure of social interaction in a multiple wearable sensor environment. We propose a search-based structure recovery method to reconstruct the social interaction structure given multiple first-person views, where each view contributes to the multi-faceted understanding of the social interaction. The proposed method is much simpler than full 3D reconstruction and suffices for capturing the spatial structure of a social interaction. The third work investigates ``presentations'', a special type of social interactions within a social group for presenting a topic. A new multi-sensor analytics framework is proposed with conventional ambient sensors (e.g., web camera, Kinect depth sensor, etc.) and the emerging wearable sensor (e.g., Google Glass, GoPro, etc.) for a substantially improved sensing of social interaction. We have conducted single and multi-modal analysis on each sensor type, followed by sensor-level fusion for improved presentation self-quantification. Feedback from the presenters shows a great potential for the use of such analytics. At the same time, we have recorded a new multi-sensor presentation dataset, which consists of web cameras, a Kinect depth sensor, and multiple Google Glasses. The new dataset consists of 51 presentations of varied duration and topics.

To sum up, the three works have explored the social interaction from ambient sensor environment to wearable sensor environment; generic spatial structure of social interaction to special type of social interaction ``presentation''. In the end, the limitation and broad vision for social interaction analysis in multi-sensor environment are discussed.