Imparting Human Objectives to Robots: Goal Specification from Language, Sketches, and Demonstrations
COM1 Level 3
MR1, COM1-03-19
Abstract:
Robots are increasingly capable of acting autonomously, but they still struggle to act on human intent. People express objectives through language, sketches, or demonstrations, which are natural for humans but not directly usable by robots. This thesis frames goal specification as an interface problem between humans and robots: how to design the modality, representation, and translation process that make human intent actionable for robot decision-making.
For explicitly expressed objectives, I develop structured interfaces that ground language and sketches into precise robot-executable goals. SetItUp translates natural language instructions for functional object arrangement into symbolic scene graphs and physically feasible object poses. StackItUp translates hand-drawn sketches into relational graphs and stable 3D structures using stability-aware diffusion models. For implicitly expressed objectives, where intent is available only through demonstrations, I study inverse reinforcement learning as the interface between human behavior and robot execution. I show that planning horizon is a key structural prior for making reward inference scalable and generalizable. Receding Horizon IRL uses local horizon structure for efficient reward inference, while Effective Horizon IRL analyzes how planning horizon affects overfitting and generalization in low-data regimes.
Together, these works argue that goal specification is not about finding a universal representation of intent, but about designing task-appropriate interfaces that make human objectives natural to express, tractable to infer, and effective for robot execution.