CMU's Pixel-UMI Trains Mobile Robots From Handheld Demos at 87% Success

Carnegie Mellon researchers have introduced Pixel-UMI , a system that converts demonstrations recorded with a sensorized handheld gripper into training data for a wheeled robot with a torso and two arms. The accompanying paper reports 87.0% average task progress across five mobile-manipulation tasks. The metric rewards completed stages within a task, including runs that stop before full completion.

Handheld recording lets operators collect demonstrations without occupying a robot or building a teleoperation station for every worker. Pixel-UMI addresses the difficult transfer step: fitting human head and hand trajectories to a robot whose body, joints, cameras, and control delays differ from the demonstrator’s.

Human motion meets robot geometry

The project extends the Universal Manipulation Interface , or UMI, introduced by Chi and collaborators. UMI records demonstrations through handheld grippers and represents actions as relative trajectories, while matching policy timing to the target hardware’s latency. That design has worked well for tabletop, single-arm manipulation.

Mobile manipulation adds a wheeled base, torso, second arm, head camera, and more opportunities for collision. When a person reaches into a refrigerator, their shoulders, elbows, and head follow paths that a specific robot cannot reproduce joint for joint. Directly mapping the recorded head and gripper poses can drive the robot into doors, counters, boxes, or chairs.

Pose errors also change the camera views available during deployment. A policy trained on images from the demonstrator may receive substantially different wrist or head images once the same trajectory has been fitted to the robot, weakening the connection between visual observations and learned actions.