The Effects of Gesture-based Interaction on User Behavior: An Embodied View
Dr Chan Hock Chuan, Associate Professor, School of Computing
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
Gesture-based interaction, which allows users to control devices with hand or finger motions, is becoming increasingly popular in the past few years. Although it has garnered much interest from practitioners and academics, many extant gesture-based interaction studies are from the technological and usability perspective. Drawing on the embodied cognition theory, this thesis focuses on the psychological and behavioral impacts of gesture- based interaction and investigates how it shapes user judgment and behavior via two mechanisms: facilitating mental imagery and activating experiences associated with a specific gesture. These constitute two essays of the dissertation.
The first essay examines the impacts of gesture-based interaction on virtual product experience. Specifically, we compared three types of interaction modality, i.e., mouse, touchscreen gesture-based interaction and mid-air gesture-based interaction, and their ability to facilitate haptic and spatial imagery. We found that touchscreen gestures performed best in terms of eliciting haptic imagery while mid-air gestures performed best in terms of eliciting spatial imagery when appropriate product presentation format was adopted. In addition, mental imagery further reduced cognitive uncertainty and improved emotional attachment, thus leading to a better virtual product experience.
The second essay investigates the effects of gesture-based interaction on users' self-regulation in the pursuit of a healthy lifestyle. Specifically, we compared two types of touch mode, i.e., hard press and gentle tap, which are believed to trigger different thinking and thus result in different decision and behavior. We found that inducing users to press (vs. tap) during mobile interaction can improve their self-regulation in terms of selecting healthy food (experiment 1), setting higher goals, and taking more physical exercise (experiment 2). In addition, such effects were more salient among users with higher health awareness and a promotion-focused health orientation.
Overall, the thesis highlights that it is important to incorporate bodily actions and physical feelings into future HCI research. In particular, bodily actions (e.g., gestures) could not only be used as an approach to control the digital system, but also be used as a source of information by users that affects their judgment and behavior. Thus, gesture-based interaction could be strategically designed to facilitate cognitive activities and behavioral change. This dissertation also offers valuable contributions to literature on mental imagery (e.g., differentiating spatial imagery and haptic imagery) and embodied cognition theory (e.g., identifying the boundary conditions of embodiment effects).