Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision. The supervised approach of imitation learning can handle short tasks but suffers from compounding errors and the need of many demonstrations for longer and more complex tasks. Reinforcement learning (RL) can find solutions beyond demonstrations but requires tedious and task-specific reward engineering for long-horizon, multi-step problems. In this work we address the difficulties of both methods and explore their hierarchical combination. To this end, we propose HRL-BC, a hierarchy of RL policies operating on pre-trained skills, that can learn long-horizon manipulations using no intermediate rewards and no demonstrations of full tasks. Our method allows efficient training of basic skills from few visual demonstrations by taking advantage of recent CNN architectures and data augmentation. Combining our hierarchical approach with the learned basic skills, we manage to learn long-term policies for manipulation tasks such as making a simple breakfast.

The policies have been trained using MImE, a simple interface based on pybullet simulator that provides tools to create complex manipulation tasks with a UR5 robotic arm. It is composed of 4 environments ranging from simple to more complex tasks: UR5-Pick, UR5-Pour, UR5-Bowl and UR5-Breakfast.