Act to See and See to Act: A Robotic System for Object Retrieval in Clutter

Abstract:

Object Retrieval in Clutter is an extremely challenging task for the state-of-the-art robotic system to perform. The challenges come from the incomplete knowledge of the environment. On one hand, a robot has imperfect sensing due to occlusion among objects. On the other hand, a robot needs to physically interact with objects of unknown physical properties.

Inspired by humans, we adopt the strategy of Act to See and See to Act to equip a robot with better sensing and control capabilities. We propose a robotic system that can reliably retrieve objects in clutter under uncertainties in sensing due to occlusion and control due to unknown objects' physical properties.

To alleviate uncertainties in sensing, we model the problem of object search in clutter as a POMDP to handle perceptual uncertainty. The proposed planner was able to select near-optimal actions to remove occlusion and reveal the target object efficiently. The enabling factor for efficient search algorithm lies in the exploitation of the spatial constraints of the problem. We concluded that POMDP planning is effective for problems which require multi-step lookahead search.

To handle uncertainties in control, we adopted a learning approach. We devised Push-Net, a deep recurrent neural network, to push an object from one configuration to another robustly and efficiently. Training with a variety of objects with varying physical properties enables Push-Net to push novel objects robustly. We also show that embedding physical understanding (center of mass) about objects in Push-Net helps select more effective push actions.

We aim to integrate both the POMDP planner and the Push-Net into a single robotic system for object retrieval in clutter. We describe the system architecture and identify technical gaps with proposed solutions.