Learner-AI Interaction in AI Tutoring: A Two-Study Investigation of Over-reliance and Logical Reasoning Development
COM1 Level 3
MR1, COM1-03-19
Abstract:
The rapid advancement of artificial intelligence (AI) has driven the widespread adoption of AI-based learning tools in educational contexts. While these tools hold great promise, many are developed without grounding in established pedagogical frameworks and often deliver generated instructional content in a passive and unsolicited manner. This has raised concerns among educators and researchers about the risk of learners becoming overly reliant on AI-generated materials. Yet, there is little empirical research that directly examines whether such over-reliance occurs during learner-AI interactions and how it affects actual learning outcomes. This thesis addresses this gap by investigating the prevalence and effects of AI over-reliance in AI-based learning environments and by exploring design strategies to mitigate these risks.
Study I explores the impact of AI-generated learning materials (i.e., AI tutoring) on learners’ over-reliance and logical reasoning development. Through a laboratory experiment, we found that learners often uncritically accepted AI-generated tutoring content, regardless of its accuracy. Notably, higher levels of over-reliance were associated with greater declines in reasoning development. Additionally, we explored the learning mechanism by examining metacognitive processes during interactions with AI. Our analysis revealed a placebo effect wherein AI tutoring reduced self-monitoring accuracy and resulted in an illusion of knowledge. Furthermore, our eye-tracking data analyses identified an inverted U-shaped relationship between attention to task content and reasoning development, suggesting that learners who maintained a balanced focus between the task and AI tutoring achieved optimal learning outcomes.
Study II proposes and examines an on-demand AI access modality designed to promote active engagement and enhance metacognitive monitoring. In a controlled experiment involving 203 learners assigned to on-demand AI tutoring (n = 68), always-available AI tutoring (n = 68), or a no-AI control condition (n = 67), the on-demand design effectively attenuated AI over-reliance. However, its impact on reasoning development was heterogeneous. High-ability learners demonstrated meaningful improvements, whereas low-ability learners showed limited gains relative to the other two conditions. Further analyses of metacognitive monitoring revealed distinct calibration dynamics underlying this divergence: high-ability learners adjusted their monitoring based on accumulated experience, while low-ability learners placed disproportionate weight on recent feedback. Notably, the on-demand design amplified these differences, facilitating effective calibration for high-ability learners but not for those with lower ability.
Taken together, the thesis yields meaningful implications for both the study of learner-AI interaction and AI over-reliance in learning contexts. It also provides practical insights into designing effective and meaningful AI tutoring systems.