Graph Inverse Reinforcement Learning from Diverse Videos

Research on Inverse Reinforcement Learning (IRL) from third-person videos has shown encouraging results on removing the need for manual reward design for robotic tasks. However, most prior works are still limited by training from a relatively restricted domain of videos. In this paper, we argue that the true potential of third-person IRL lies in increasing the diversity of videos for better scaling. To learn a reward function from diverse videos, we propose to perform <PRE_TAG>graph abstraction</POST_TAG> on the videos followed by temporal matching in the <PRE_TAG>graph space</POST_TAG> to measure the task progress. Our insight is that a task can be described by entity interactions that form a graph, and this <PRE_TAG>graph abstraction</POST_TAG> can help remove irrelevant information such as textures, resulting in more robust reward functions. We evaluate our approach, GraphIRL, on cross-embodiment learning in X-MAGICAL and learning from human demonstrations for real-robot manipulation. We show significant improvements in robustness to diverse video demonstrations over previous approaches, and even achieve better results than manual reward design on a real robot pushing task. Videos are available at https://sateeshkumar21.github.io/GraphIRL .