Peking University's PhysAg3D Turns Single Photos Into Moving 3D Robot Assets
PhysAg3D generates physically interactive 3D assets from a single image, with joints and soft bodies ready for simulation Pairs a frozen image-to-3D mesh generator with a vision-language agent that reasons about parts and interactions Hierarchical representation links interaction intent to joints
Takeaways
- PhysAg3D generates physically interactive 3D assets from a single image, with joints and soft bodies ready for simulation
- Pairs a frozen image-to-3D mesh generator with a vision-language agent that reasons about parts and interactions
- Hierarchical representation links interaction intent to joints to concrete mesh bindings
- Reports 91.3% Action Completion Rate and 88.0% stability pass on 50 real-world objects
- Handles both rigid articulation and compliant deformation like headphone pads
- Code and 223 assets available via GitHub and the project page
Single-image 3D generators can produce detailed, textured meshes, but those assets usually lack instructions for movement and deformation. Their geometry cannot specify that a printer tray slides, a chair reclines, or a headphone pad compresses.
Researchers at Peking University, Huawei, and Wuhan University developed PhysAg3D to add that missing behavior. Given one image, the system generates a textured mesh, identifies interactive parts, constructs joints or deformable regions, and binds those controls to the relevant surfaces. The resulting asset is prepared for embodied AI and robotics simulators.
A mesh cannot tell you what moves
Image-to-3D systems such as TRELLIS, Hunyuan3D, and Rodin can generate closed, textured surfaces from a photograph. Visual reconstruction alone leaves several questions unanswered: which components move, what type of motion they support, where the pivot lies, and how far each part can travel.
Robotics simulations and manipulation datasets need that semantic and physical structure. Existing systems such as PhysX-Omni and aDSL integrate articulation into the generative pipeline. The PhysAg3D authors argue that this coupling can limit either visual fidelity or the range of physical behavior.
Two specialists build one asset
PhysAg3D separates appearance generation from interaction construction. A conventional 3D generator creates the textured mesh, while a vision-language agent analyzes the same image to infer parts, possible actions, and physical properties. A hierarchical representation connects those inferences to the generated surfaces.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
