NVIDIA's LucidLyra Turns a Single Photo Into a Fully Labeled 3D Scene

Single-image 3D generators can create plausible scenes, yet their outputs often lack object identities and semantic labels. A camera can move through the reconstruction, but an application cannot reliably identify the couch, select one car, or locate the drivable surface.

LucidLyra , from NVIDIA’s Spatial Intelligence Lab, adds those capabilities during reconstruction. The research framework converts one image into a 3D scene containing appearance, geometry, language-aligned semantic features, and object-instance embeddings. It supports novel-view rendering, text or click selection, and 3D object masks without per-scene optimization.

A scene applications can query

LucidLyra represents each scene with 3D Gaussians, small spatial primitives that can be rendered efficiently from new camera positions. Alongside color and geometry, its Gaussians carry two kinds of learned features: semantic features associate regions with open-vocabulary text prompts, while instance features group points belonging to the same object.

That combination addresses a gap between two established research areas. Image-to-3D generators produce visually plausible environments, while 2D foundation models such as SAM and SigLIP label pixels and connect images to language. Existing methods often join those systems in a costly second stage, depend on captured multiview data, or optimize a representation separately for every scene.

Robotics, simulation, and scene-editing tools need geometry and labels in the same coordinate system. A robot planner, for example, needs to distinguish a floor from an obstacle while retaining enough visual detail to render the environment from its own camera.