Zhao-Heng Yin, Sherry Yang, Pieter Abbeel
The paper introduces an object-centric 3D motion field representation for extracting actionable insights from human videos to improve robot learning, achieving significantly better performance in real-world tasks compared to prior methods.
Researchers are exploring ways to teach robots by observing human actions in videos, but capturing the necessary details from these videos is challenging. This study proposes a new way to represent actions using a '3D motion field' that focuses on objects, which helps robots learn more effectively from human demonstrations. The approach involves a novel training method to accurately capture object movements, even when video quality is poor, and a prediction model that helps the robot apply what it learns to different situations. Tests show that this new method significantly improves the robot's ability to understand and replicate human actions, even in complex tasks.
