Robot Manipulation via Action Decomposition and Composition

The ability to efficiently acquire generalized skills from demonstrations and apply them across diverse real-world scenarios is a central challenge in robot manipulation. Unlike conventional robot learning tasks that rely on extensive action demonstrations for single-task performance, zero-shot manipulation demands that robots leverage multiple learned skills to accomplish novel tasks. In this work, we propose an action decomposition and composition framework designed to efficiently transfer foundational manipulation skills to various new derivative tasks. Our approach first decomposes a demonstration into fundamental skills such as foundation movement and rotation, which can then be recombined and applied to a new task not present in the initial training. Using an action prediction model, we generate potential manipulation poses, including trajectory poses and other potential interactive poses, for each subtask within the derivative task, guided by both robot actions and video frames captured from the robot’s cameras. To ensure accuracy, we further refine these generated poses by filtering out misleading actions and selecting the most probable manipulation poses to guide the robot effectively. Experimental results on hotdog preparation task demonstrate that our framework can successfully enable both robotic arms and automated guided vehicles to perform derivative tasks, showcasing its versatility and efficiency.