Heads-up Conversational Authoring: A Perception-Interaction-Cognition Framework for Mobile Content Creation
COM1 Level 3
MR1, COM1-03-19
Abstract:
Mobile computing has evolved far beyond simple messaging, with increasingly complex productivity tasks becoming routine on smartphones even when users are away from their desktops. However, these tasks remain constrained by the “heads-down” posture and limited input/output (I/O) bandwidth of conventional mobile paradigms. While the synergy of voice interaction, large language models (LLMs), and optical see-through head-mounted displays (OHMDs) enables the Heads-up Conversational Authoring paradigm, it introduces a new shift in challenges: although it alleviates the physical conflicts of mobile I/O, it imposes new platform constraints and cognitive demands. This thesis deconstructs these challenges through a three-layer framework — Perception → Interaction → Cognition — to bridge the gap between low-level physical constraints and high-level metacognitive partnerships.
We address these challenges through three interconnected projects. First, at the Perception Layer, TextSpacing establishes a visual foundation by quantifying the trade-offs between text readability on OHMDs and mobility, ensuring efficient content consumption on the go. Second, at the Interaction Layer, GlassMail validates the feasibility of a voice-based mobile email creation workflow, yet reveals that fluid interaction mechanics alone cannot sustain complex authoring without deeper cognitive support. Finally, at the Cognition Layer, we explore the broader implications and high-level trade-offs of this paradigm shift relative to current practices by developing a design probe, GlassMail+. Our empirical results show that while this approach sustains user confidence and reduces mental load during mobility, in-the-wild deployment highlights that it also imposes high Intent Management Overhead and a significant Granularity Gap, ultimately leading to a loss of structural context. This finding identifies an inherent cognitive bottleneck for reactive scaffolding as task complexity and environmental dynamism increase, providing the empirical foundation that motivates future research on proactive intent management.
The primary contribution of this thesis is the conceptualisation, empirical validation, and boundary identification of the Heads-up Conversational Authoring paradigm. By progressing from physical constraints to high-level intent management, this work provides a comprehensive roadmap for designing wearable AI assistants that act not merely as tools for execution but as capable partners in a balanced cognitive partnership.